Collecting Elasticsearch based search-related MustGather data for Query runtime issues
Prepare the following MustGather information before you contact HCL Support to help with Elasticsearch issues.
Before you begin
This MustGather will make use of request-level tracing to only apply tracing for the Query REST request being used for the scenario, rather that applying service-level tracing which would apply tracing for all Query REST request being processed in the application. To enable request-level tracing, follow the steps in the Logging and troubleshooting the Ingest and Query services.
About this task
This MustGather can be used to investigate the cause of general Elasticsearch-based runtime issues. If you are trying to answer one of the following questions, this is the MustGather to use:
- Why am I getting a different number of search results than expected?
- Why are my search results ordered in a different way than expected?
Procedure
- Confirm the Query REST request that applies to the scenario you are investigating. If you aren't sure what that is, and you are using a React-based storefront, you can use your browser's Developer Tools network tab when navigating the storefront to reproduce the issue, and view the Query REST call or calls being made.
- Execute the Query REST request with an additional header "X-Log-Level=TRACE". Do not add this to your REST request through a URL parameter.
-
Collect all the trace files generated in the Query service:
/app/ESQueryService/logs/
What to do next
- Reviewing the initial request
- The start of the scenario is going to be processing of the incoming REST
request to the Query service. The resource handler used will be based on
the API invoked. For example, when performing a search on the storefront
for "couch", this will make use of the
V2ProductResource handler due to using an
/api/v2/products API call. For
example:
2023-06-13T20:17:29.024Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.search.rest.V2ProductResource.findProducts:157 - URL: https://search:30901/search/resources/api/v2/products?storeId=11&searchTerm=couch&limit=12&offset=0&contractId=-11005¤cy=USD&langId=-1&profileName=HCL_V2_findProductsBySearchTermWithPrice - Search profile definition
- The search profile contains the main contextual information we'll use
for constructing the search query and then processing the search
results, so it is important to validate that the search profile matches
your expected behavior for this search. To validate the search profile
definition being used, you can look for the tracing output from
SearchConfigurationRegistry.getSearchProfile().
For example, here is the search profile definition for
HCL_V2_findProductsBySearchTermWithPrice being
looked
up:
If this does not match the expected profile definition, you can validate your configuration from the search profile Query API endpoint. For example, to look up the HCL_V2_findProductsBySearchTermWithPrice configuration in the Auth environment:2023-06-13T20:17:29.249Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.c.SearchConfigurationRegistry.getSearchProfile:368 - ENTRY profileName:HCL_V2_findProductsBySearchTermWithPrice resourceName:productview resourceURI:/api/v2/products?searchTerm 2023-06-13T20:17:29.249Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.c.SearchConfigurationRegistry.getSearchProfile:382 - EXIT searchProfile: SearchProfile [parentProfileName=, profileName=HCL_V2_findProductsBySearchTermWithPrice, indexName=product, query=Query [params=[{DynamicKitReturnPrice=true}], queryFields=[default.search.text, default.search.normalized, default.search.text_*, default.sku.normalized, natural.keywords.normalized], sortFields={1=manufacturer.raw asc, 2=name.normalized asc, 3=offer.* asc, 4=offer.* desc}, responseFields=[id.catentry, id.store, buyable, identifier.sku.raw, name.raw, name.override.raw, description.raw, description.override.raw, url.thumbnail, url.override.thumbnail, keyword.text, keyword.override.text, manufacturer.raw, id.member, seller.raw, type, prices.*, path.*, relationship.item.id, relationship.product.id, identifier.mpn.raw, attribute.*, kit.preconfigured, kit.model, kit.URL, kit.default_configuration, kit.parent.model, kit.configurable, kit.parent.configurable, start, end, *_display, url.seo, url.override.seo], highlight=null, spellcheck={limit=5}]]GET https://data_environment_hostname:30921/search/resources/api/v2/documents/profiles/HCL_V2_findProductsBySearchTermWithPrice - Natural Language Processing (NLP)
- Commerce Search makes use of NLP to perform analysis on the search term
or terms being used to identify intent behind the terms used and modify
the search appropriately. If you are seeing unexpected results, it may
be a result of this NLP analysis performing in a way that you may not
expect, or because it is taking into account additional data points that
you may not be considering for this search.First, to identify the NLP providers that will be used for the search, you can look for the tracing output from SearchNLPSupportProvider.invoke(). For example, here is the tracing output when using the default NLP profile:
If you want to verify that the system is using Basic NLP, then you can look for the tracing output from SearchNLPSupportProvider.invoke():2023-06-13T20:17:29.752Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSupportProvider.invoke:107 - Search NLP providers: com.hcl.commerce.search.internal.expression.provider.SearchNLPPartNumberProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPWhiteSpaceProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPCurrencySymbolProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPSpellCorrectionProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPExcludedTermProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPNumberFormatterProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPSTAExpansionProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPDependenciesParsingProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPMultiwordTermProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPLowerCaseProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPDMMProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPSpecialCharacterProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchMultiwordFilterProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPStopwordProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPWordToNumberProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPPriceFilterProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPPOSAndNERProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPColorMMProviderHelperIf you want to check that we are using Basic NLP TRACE c.h.c.s.i.e.p.SearchNLPSupportProvider.invoke:123 - Search NLP providers: com.hcl.commerce.search.internal.expression.provider.SearchNLPCustomPartNumberHelper com.hcl.commerce.search.internal.expression.provider.SearchBasicNLPCategorySearchProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPWhiteSpaceProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPCurrencySymbolProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPLowerCaseProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPSpellCorrectionProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPExcludedTermProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPFractionalNumberHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPDMMProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchMultiwordFilterProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchBasicNLPStopwordProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPPriceFilterProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchBasicNLPPOSAndNERProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchNLPColorMMProviderHelper com.hcl.commerce.search.internal.expression.provider.SearchBasicNLPBoostQueryProviderHelperIf there is a particular NLP provider that you do not want to use, or you want to include your own NLP providers, see our Natural Language Processor profiles page for more information.
Next, you will want to review each of the NLP providers to see if it made the intended effect to the search. For example, if we have a synonym (Search Term Association or STA) between couch and sofa, we can confirm that SearchNLPSTAExpansionProviderHelper picked up this synonym by reviewing the tracing output for this NLP provider:2023-06-13T20:17:30.158Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.performSTAExpansion:164 - ENTRY 2023-06-13T20:17:30.159Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.performSTAExpansion:165 - search term before STA expansion : couch ... 2023-06-13T20:17:30.529Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.getNodeData:592 - nodeData : {sofa,couch=s} 2023-06-13T20:17:30.529Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.getNodeData:594 - EXIT ... 2023-06-13T20:17:31.794Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.performSTAExpansion:468 - EXIT 2023-06-13T20:17:31.794Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.performSTAExpansion:470 - search term after STA expansion : sofa couch - Elasticsearch query
- Since HCL Commerce Search Elasticsearch queries can be cached, we
can use the tracing output from
SearchResponseCache.myInvoke() to see the
Elasticsearch query being used for our search. For example, here is the
Elasticsearch query being generated for the earlier
"couch"
search:
2023-06-13T20:17:32.468Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchResponseCache.myInvoke:562 - Cache name: com.hcl.commerce.search.internal.expression.processor.SearchExpressionProcessor Index name: auth.12001.product Source builder: {"from":0,"size":12,"query":{"function_score":{"query":{"bool":{"filter":[{"query_string":{"query":"workspace_name : \"Base\"","fields":[],"type":"best_fields","default_operator":"or", "max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50, "phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}, {"query_string":{"query":"-((type:product AND relationship.has_sku:true) OR (type:variant AND state:false)) AND (relationship.product.group:*)", "fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO", "fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true, "fuzzy_transpositions":true,"boost":1.0}},{"query_string":{"query":"id.catalog:\"11501\"","fields":[],"type":"best_fields", "default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0, "fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true, "boost":1.0}},{"query_string":{"query":"id.store:(\"11\" \"12001\")","fields":[],"type":"best_fields","default_operator":"or", "max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0, "fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}, {"query_string":{"query":"id.language:\"-1\"","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000, "enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false, "auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},{"query_string":{"query":"displayable:true", "fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true, "fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0, "escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}, {"query_string":{"query":"(*:* AND -_exists_:facets.7000000000000001010.value.raw)","fields":[],"type":"best_fields", "default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0, "fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true, "boost":1.0}}],"should":[{"query_string":{"query":"(\"sofa\" OR \"couch\")","fields":["default.search.normalized^1.0", "default.search.text^1.0","default.search.text_en_US^1.0","default.sku.normalized^1.0","natural.keywords.normalized^100.0", "natural.nouns.normalized^100.0"],"type":"most_fields","default_operator":"and","max_determinized_states":10000, "enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50, "phrase_slop":0,"lenient":true,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}, {"query_string":{"query":"(\"sofa\" OR \"couch\")","fields":["default.search.normalized^1.0","default.search.text^1.0", "default.search.text_en_US^1.0","default.sku.normalized^1.0","natural.keywords.normalized^100.0","natural.nouns.raw^100.0"], "type":"most_fields","default_operator":"and","max_determinized_states":10000,"enable_position_increments":true, "fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"lenient":true,"escape":false, "auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true, "minimum_should_match":"1","boost":1.0}},"functions":[{"filter":{"match_all":{"boost":1.0}},"script_score":{"script":{"id":"boost-script-param-1", "params":{"boostFactor":10.0,"param1":"couch"}}}}],"score_mode":"sum","boost_mode":"sum","max_boost":3.4028235E38,"boost":1.0}}, "explain":true,"_source":false,"stored_fields":["id.catentry","id.store","buyable","identifier.sku.raw","manufacturer.raw", "id.member","seller.raw","type","prices.-11005.usd","path.11001","path.11501","path.11502","relationship.item.id","relationship.product.id", "identifier.mpn.raw","attribute.*","kit.preconfigured","kit.model","kit.URL","kit.default_configuration","kit.parent.model", "kit.configurable","kit.parent.configurable","start","end","*_display","prices.list.usd","prices.offer.usd","name.11.raw", "description.11.raw","url.11.thumbnail","keyword.11.text","url.11.seo","url.12001.thumbnail","url.12001.seo","name.12001.raw", "description.12001.raw","keyword.12001.text"],"track_total_hits":2147483647, "aggregations":{"itemCount":{"cardinality":{"field":"relationship.product.group"}}}, "suggest":{"correction":{"text":"couch","term":{"field":"default.correction.text","size":5,"suggest_mode":"MISSING", "accuracy":0.3,"sort":"SCORE","string_distance":"INTERNAL","max_edits":2,"max_inspections":5,"max_term_freq":0.01, "prefix_length":2,"min_word_length":4,"min_doc_freq":0.0}}},"collapse":{"field":"relationship.product.group","inner_hits":{"name":"data", "ignore_unmapped":false,"from":0,"size":20000,"version":false,"seq_no_primary_term":false,"explain":false,"track_scores":false, "stored_fields":["url.12001.thumbnail","url.11.thumbnail","id.*","prices.*"]}}} - Final version of REST response
- To find the end of the REST request processing, you can look at the
tracing output for
AbstractSearchResource.executeSearch(). For
example, here is the final version of the REST response for the
"couch" search (condensed for
brevity):
2023-06-13T20:17:33.782Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.rest.AbstractSearchResource.executeSearch:505 - Result : <200 OK OK,{catalogEntryView=[{hasSingleSKU=false, buyable=true, resourceId=https://search:30901/search/resources/api/v2/products?storeId=11&searchTerm=couch&limit=12&offset=0&contractId=-11005¤cy=USD&langId=-1&profileName=HCL_V2_findProductsBySearchTermWithPrice, uniqueID=14033, thumbnailRaw=/EmeraldCAS/images/catalog/livingroom/furniture/chair4_b1_350.jpg, thumbnail=/hclstore/EmeraldCAS/images/catalog/livingroom/furniture/chair4_b1_350.jpg, sellerId=7000000000000003501, parentCatalogGroupID=/10501/10502, manufacturer=Stonehenge, shortDescription=Very cozy short design single sofa., catalogEntryTypeCode=ProductBean, groupingProperties={groupOfferPriceRange=[Ljava.lang.Object;@682c66e4, groupListPriceRange=[Ljava.lang.Object;@c25e4f60, groupCount=4, groupOwner=14033, groupMaxPriceValue=749.99, groupHero=14035, groupMinPriceValue=749.99}, name=Stonehenge UltraCozy Single Sofa, partNumber=LR-FNTR-0004, storeID=11, seo={href=/stonehenge-ultracozy-single-sofa-lr-fntr-0004}, price=[{usage=Display, description=L, currency=USD, value=800.0}, {usage=Offer, contractId=-11005, description=I, currency=USD, value=749.99}], attributes=[{identifier=Color, attribute.group= , attribute.natural=["Brown","Dark Grey","Seaweed","Denim"], usage=Defining, values=[{sequence=[2.0, 10.0, 10.0, 10.0], identifier=[brown, darkgrey, seaweed, denim], unitOfMeasure=[one, one, one, one], unitID=[C62, C62, C62, C62], image1=[/EmeraldCAS/images/catalog/swatches/sw_brown.png, /EmeraldCAS/images/catalog/swatches/sw_darkgrey.png, /EmeraldCAS/images/catalog/swatches/sw_seaweed.png, /EmeraldCAS/images/catalog/swatches/sw_denim.png], value=[Brown, Dark Grey, Seaweed, Denim], image1path=[/hclstore/EmeraldCAS/images/catalog/swatches/sw_brown.png, /hclstore/EmeraldCAS/images/catalog/swatches/sw_darkgrey.png, /hclstore/EmeraldCAS/images/catalog/swatches/sw_seaweed.png, /hclstore/EmeraldCAS/images/catalog/swatches/sw_denim.png], uniqueID=[7000000000000003002, 7000000000000003010, 7000000000000003014, 7000000000000003020]}], displayable=true, merchandisable=true, searchable=true, sequence=1.0, storeDisplay=false, name=Color, attribute.store=12001, facetable=true, comparable=true, uniqueID=7000000000000000501, swatchable=false}]}, ... ], displayable=true, merchandisable=true, searchable=true, sequence=1.0, storeDisplay=false, name=Color, attribute.store=12001, facetable=true, comparable=true, uniqueID=7000000000000000501, swatchable=false}]}], recordSetComplete=false, recordSetCount=12, recordSetStartNumber=0, recordSetTotal=22, resourceId=https://search:30901/search/resources/api/v2/products?storeId=11&searchTerm=couch&limit=12&offset=0&contractId=-11005¤cy=USD&langId=-1&profileName=HCL_V2_findProductsBySearchTermWithPrice, recordSetTotalMatches=22, resourceName=productview, facetView=[], metaData={price=1}},[]>