Release changes to the Query service

Version 9.1.18.0 Query service changes

Zookeeper through the API: Ingest configuration data in zookeeper nodes will not be initialized at startup. It will only store custom overrides of Ingest Configuration in Zookeeper. The GET API will return the default ingest-config.json from the resources and merge the Zookeeper custom overrides (if available).
Search result PLP displays Product URLs: Term search will return the SEO URL for the product (before it was item). To maintain the previous UI experience, a new parameter named selected has been introduced to the response under the attribute field for descriptive attributes, which can be selected by storefront.
In Extended navigation, products will be sequenced using global_sequence: In the case of extended navigation, products will be sequenced using global_sequence. This updated the sorting method to global_sequence.catalog-catgeory.sequence
Added STA threshold validation in config API: The Configuration STA API (e.g., /configuration?nodeName=product_sta) is used primarily during the ingest process to load STAs (Storefront Attributes). This means that any errors caused by exceeding the maximum STA limit will only occur during the ingest process, not when a merchandiser creates STAs using the CMC (Commerce Management Center).; Recommendation: Merchandisers should create and manage STAs through the CMC interface, as it does not directly interact with the Configuration STA API. This API endpoint is for internal use only.

Version 9.1.17.0 Query service changes

New parameters for termDropping: Introduced new parameters for termDropping, including termDropOperator for the default operator and searchPerformed for the actual search term used in the final search. This parameter in the metadata can be used to determine the exact phrase on which the search is performed following a fall back or term dropping.
PLP category navigation with facet: PLP category navigation with facet will return SEO URL for product.

Version 9.1.16.0 Query service changes

Basic Natural Language Processing (NLP) Search rule: The CTRL_PARAM_SEARCH_INTERNAL_ESPOT_NAME is not configurable.

Version 9.1.15.2 Query service changes

Classification change in Basic NLP profile: We identified the classification category provided in the basic NLP profile. Based on this information, set the parameter CTRL_PARAM_INTERNAL_PRESENT_IN_BASIC_NLP_PROFILE_CLASSIFICATION and boost the search for that category in search by term call.

Disabling NLP: To disable NLP completely we have introduced a new ingest global variable - "flow.disable.basic.nlp". If set to true, will ignore NLP processing in Nifi for Basic NLP. For Advance NLP, this new variable will have no impact.

Search containing currency names: Added “currency.properties” file under resources. On searching currency term, if the search term is a single term (currency name), it will NOT be ignored, and the number of results will be non-zero; otherwise, multi-word search terms will be ignored.

Color swatches for items in product card: Added two flags in the wc-component, "ReturnUnavailableAttributeValueForBrowsing" and "ReturnUnavailableAttributeValueForKeywordSearch" to get color swatches for keyword search and color SKU filter.

Result for Multiword search terms with and without hyphens: NER_TAG_HYPHENATED and nlp.enable.hyphenated.ner: The HyphenatedSTA provider is responsible for parsing your search term and applying replacements based on NLP. This allows the seamless handling of search terms containing hyphenated words, both with and without hyphens, enhancing search accuracy and user experience. The configuration parameter nlp.enable.hyphenated.ner controls the activation of hyphenated STA support. By default, this parameter is set to true.

Result for product not a part of catalog: Product must not get return back if it is not part of the catalog. In category related fields, the CATALOG_ID_PLACE_HOLDER Id is added so, category can be matched by catalog.

Deep Sequencing - product and category sequence

ENABLE_DEEP_PRODUCT_SEQUENCE: Deep search sequencing refers to the process of ordering products within a category and all its subcategories. This process is used to enhance the storefront by using Category-based browsing flows that customers can use to browse products.
enableSearchTermDropOperator: When the enableSearchTermDropOperator flag is enabled and no results are found for a requested search term, the system initiates term-dropping based on the NLP profile “termDroppingPriority” section.

Added "overrideFieldName" : "url.thumbnail,url.seo" under hero inside searchProfile HCL_V2_findProductsByCategoryWithPriceRange. The product thumbnail should be for the variant that matches the filter selections.
Added “flow.index.flattened” ingest configuration: flow.index.flattened has been introduced to avoid map explosion from occurring. However, this option degrades performance during faceting and filtering, particularly when the facetable attributions include a large number of entries. The default option is "false", which means do not index using the "flattened" data type. Only use this option (by setting it to "true") if you experience exceptionally long ingest times while loading a large number of attributes into a product page.
Added nlp.classification.write.to.ner.file in wc-component. After removing the classification from the value using the configuration endpoint, no entries will be added to the custom NER file corresponding to that value.

Version 9.1.15.1 Query service changes

Change in HCL_findNavigationSuggest_Products: While using HCL_findNavigationSuggest_Products profile, we were retrieving process prices and short descriptions for products in auto-suggestion. It improved performance as it was also taking some extra time in price processing logic. As per requirement, the short description field has been removed from this profile.

Inner hits sorting for color matchmaker and search rule: CTRL_PARAM_INTERNAL_SEARCH_RULE_EXIST_FOR_TERM = "_wcf.search.internal.rule.exist.for.term": Initially, inner hits sorting of SKUs (inside collapse section) on the basis of SKU sequence was fixed. Later, we found out that when we do this fix, it affects the inner hits sorting order for color matchmaker and search rules and provides preference to SKU sequence. To fix this, if any search rule or color matchmaking is applied to the search term, in that case, it will not change any sorting order. If the search rule/color matchmaker is not applied on the search term, then inner hits sorting on the basis of SKU sequence will be applied.

Introduced flag flow.dataload.category.syncProducts.

Version 9.1.15 Query service changes

Change in search profile

Change was done because when searching for a product part number, it was not returning the PDP for the default SKU.When changing the sequence of the SKUs of a product out of the box, tt was returning random SKUs instead of the default one.
Added "relationship.product.sequence" field.
Added "sort" : "relationship.product.sequence asc" in the group section for the following profiles:
- HCL_findProductsBySearchTermWithPrice.
- HCL_V2_findProductsBySearchTermWithPrice.
- HCL_V2_findProductsBySearchTerm.
- HCL_findProductsBySearchTerm.
- HCL_V2_findProductsBySearchTermRanking.
- HCL_findProductsBySearchTermRanking.

Changes in Basic NLP 'nlpParsing'

Add 'search-rule' evaluation in parsing information.
Rather than including all parsing information under 'nlpParsin', we are updating it as 'searchExecution', which provides a comprehensive overview of the parsing details for search terms. Below is the classification group to clarify which processing is part of NLP:
- SearchTermParsingMetadata (non-nlp parsing)
  - searchTerm
  - excludeTerm
  - spellCorrect
  - stopword
  - sta
  - partNumber.
- NlpParsingMetadata
  - priceFilter
  - pos
  - ner
  - uom
  - fraction
  - dmm
  - color.

Non-displayable attributes for Elastic Search: Added new flag ReturnNonDisplayableAttributes. To enable non-displayable attributes in API response, ReturnNonDisplayableAttributes property should be set with “true” value. By default, non-displayable attribute will not be returned in response as its default value is false in the wc-component.

Version 9.1.14.1 Query service changes

Changed to enable override on description/keywords in Keyword Autosuggestion for Esite model.
Introduced ProductAttributeValueFilterWithSKUAttributeValue.
Introduced SearchCommerceTokenHelperUtil/contractIdsCheckNotRequired.

Version 9.1.13 Query service changes

Response order of Product suggestion API: To get the same result sequence for product suggestions, use profile HCL_V2_findNavigationSuggestion_Products for the configuration property SearchBasedProductSuggestionProfile. This can be added through the configuration endpoint in zookeeper. Its default value is null.

Version 9.1.7 Query service changes

i. Lookup Profile

Lookup search profiles are applicable only when grouping is enabled. Previously there were hardcoded search profile name code references. Overriding those search profiles resulted in the expression of the custom behavior in all scenarios referencing the profile.

For example, previously HCL_findCatalogEntryById was hardcoded for both term search and browse scenarios. If this search profile was overrideen, the new behavior was expressed in both term search and browsing scenarios with no way to isolate customization. The lookup profile was introduced to allow a profile override with any name. The code will check for the lookup profile first, instead of using the hardcoded profile.

This functional enhancement introduces a new optional lookupProfileName attribute in the search profile. The custom override profiles will still work as-is. See the following custom search profile example that references a lookup profile:

Create a custom profile

To find a specific inventory item, add a response field in the custom profile that specifies the inventory number as inventories.10501.quantity.

{
	"parentProfileName": "HCL_findProductsBySearchTerm",
	"profileName": "X_findProductsBySearchTerm",
	"lookupProfileName": "X_findCatalogEntryById",
	"query": {
		"responseFields": [
			"inventories.10501.quantity",
			“workspaceName”
		]
	}
}

Create a lookup profile

{
			"parentProfileName": "HCL_findCatalogEntryById",
			"profileName": "X_findCatalogEntryById",
			"query": {
				"responseFields": [
					"inventories.10501.quantity"
				]
			}
}

For more information about lookup profiles, see Inventory and custom fields in a custom search profile.

ii. V2 search profiles

In previous versions Search used the same search profiles for both Aurora and React (Emerald/Sapphire) storefronts. This resulted in the same behavior for both Aurora and React storefronts when a profile was overridden. In Version 9.1.7, search profiles were segregated into V1 and V2 profiles to align with the V1 and V2 Query Service REST API. The V2 search profiles are intended for use with the React storefronts, which require the V2 REST API. The V1 search profiles are used with the V1 REST API.

If you are using the V2 REST API, it is recommended that you use V2 search profiles so as to stay in alignment with the evolution of the V2 REST API.

You can find sample V2 profiles in Search profiles.

iii. Access control for the Query Service configuration endpoints

Authentication has been introduced to protect data-query configuration (admin) endpoints. The data-query service has configuration endpoints while auth-query and live-query only serve the storefront REST API endpoints. Auth-query and live-query serve the the storefront REST APIs for browsing for the shoppers and these API endpoints do not require access control.

Note: When browsing endpoints are accessed through auth-query and live-query, there is no impact from migration point of view. However, accessing configuration API endpoints through data-query now require an authentication header with SPIUSER and password in the request.

Version 9.1.8 Query service changes

Natural Language Processing (NLP) Profiles: A new NLP Profile feature has been introduced in 9.1.8 to provide a method for controlling the preprocessing flow of search terms before executing an Elasticsearch query. These NLP Profiles can be created at store level.; Prior to Version 9.1.8, complete logic for NLP, such as the PartNumber, CurrencySymbol, DMM, Color Matchmaker etc., were present in the com.hcl.commerce.search.internal.expression.provider.SearchNLPSupportProvider class. This logic has been moved to separate helper classes and externalized in the form of NLP Profiles to facilitate customization in the extension framework.
Note: There is no impact on migration due to this functional enhancement as this is internal refactoring of code. It does not change the existing behavior of NLP.
For more information on NLP profiles, see Extending Natural Language Processor profiles.

Upgrading to Version 9.1.12 Query service

When Upgrading to HCL Commerce Search Version 9.1.12.0, perform a full re-indexing in order for the new Search Term Association (STA) function to re-analyze the STA terms correctly.