Release changes to NiFi
Changes to NiFi connectors have been made in recent releases. These are described in descending order from the most recent.
9.1.16.0 release changes
- Auth and Live Index Separation Related support when not using push-to-live involves following listed changes
-
- DeleteLink process group has been added in Reindex connector
- Removed redundant RefreshLink process group right before ReplicateLink from Reindex connectors
- Moved AliasLink process group to the very end in Reindex, Alias, Push-to-live, Update-live connector
- Added ReplicationLink to Update-live connectors
- Added replica variable for deployment configuration Reindex, Alias, Push-to-live, Update-live connectors
- Added support for the live.store.master index
- Elasticsearch reindexes performance tuning-related changes
-
- Removeindex.shard.size and index.store.type from all connector definitions and use the default value defined in the flow instead in Reindex, Rank connectors
- Added keep.source.index option to MoveLink process group in Reindex, Rank, Push-to-live, Update-live connectors
- Use cluster.index.shard.allocation attribute to define the target environment node attribute when moving shards
- Added support for Shrink operation
- Added RenameLink to support dynamic auth or live separation indexing in Reindex connector
- Added dynamic auth or live separation indexing support to live Reindex connectors
9.1.15.2 release changes
- ElasticSearch Schema
-
- Added dynamic template for facets.normalized,facets.raw,facets.key,global_sequence_catalog_category, and default.search.path_name_<locale> in both cas_product_elastic_schema.json and product_elastic_schema.json.
- Changed the types of facets and filtersfields to conditional based on the flow.index.flattened ingest configuration in both cas_product_elastic_schema.json and product_elastic_schema.json.
- Stopped copying path_name.string into the default.search.text field in both cas_product_elastic_schema.json and product_elastic_schema.json.
- Stopped copying name.store.text, description.store.text and keyword.store.normalized into the default.suggestion field in cas_product_elastic_schema.json.
- NiFi Pipeline Modifications
-
- Inter changed position of
ProductStage1b(EnrichDocument)[DatabaseCASProductStage1b]
withProductStage1h(FindChildItems)[DatabaseCASProductStage1h]
in uth.product.cas and live.product.cas. - Added new connectors
auth.alias.cas,live.alias.cas
andauth.colorfamily
.
- Inter changed position of
9.1.15.1 release changes
- ElasticSearch Schema
-
- Added dynamic_template keyword.<store>.normalized and updated the field default.correction type from object to text in cas_product_elastic_schema.json.
- Added field keyword.normalized and dynamic_template keyword.override.normalized in product_elastic_schema.json.
- Added custom analyzer custom_<locale>_stemmer_override in both cas_product_elastic_schema.json and product_elastic_schema.json.
- NiFi Pipeline Modifications
-
- Added
Override Stemmer[OverrideStemmer]
andWaitLink - Product Stage 1a[WaitLink]
afterWorkspace Schema[WorkspaceSchema]
inauth.reindex
,auth.reindex.cas
,live.reindex
,live.reindex.cas connector
. - Removed
Cache Invalidation - Product[InvalidateProduct]
from all inventory pipelinesdataload.product.cas
,auth.inventory
,live.inventory.cas
,live.inventory
. - Moved
CASSynchronizeCategories [Synchronize Categories]
,DatabaseCASCategoryStage1a [Category Stage 1a (Main Document)]
,DatabaseCASCategoryStage1b [Category Stage 1b (Find Facets)]
,SendCategoryToES [Send Category to Elasticsearch]
afterWaitLink - Product Stage 1a[WaitLink]
and removedWaitLink - Category Stage 1a[WaitLink]
in thedataload.product.cas connector
.
- Added
9.1.15 release changes
- ElasticSearch Schema
- Addition of fields in log_dynamic_mappings.json:
Added fields history and language in log_dynamic_mappings.json to include change history event IDs in the run response.
- New filters and custom analyzer in
cas_product_elastic_schema.json:
Introduced the pattern_replace_char_filter and whitespaces_replace_char_filter filters.
Implemented a custom analyzer named es_pattern_replace for the sku and mpn fields in cas_product_elastic_schema.json.
- Addition of fields in log_dynamic_mappings.json:
- NiFi Pipeline
-
- Added WaitLink - Product Stage 1a[WaitLink],
WaitLink - Enrich NLP[WaitLink], and moved
Enrich NLP[NLPStage1] after
WaitLink - Product Stage 1g[WaitLink] in
the
dataload.product.cas
connector. - Added WaitLink - Category Stage 1a[WaitLink],
moved Clear TI Delta
Table[DatabaseClearTIDeltaTable] before
WaitLink - Enrich NLP[WaitLink] in
auth.reindex.cas
,live.reindex.cas
connectors. - Added WaitLink - Flow Control[WaitLink] and
ContinueLink - Flow Control[ContinueLink]
in
auth.validate
,live.validate
connectors. - Added SplitLink[SplitLink] and
WaitLink - SplitLink[WaitLink] in
auth.reindex.cas
,live.reindex.cas
connectors. - Moved WaitLink - Product Stage 1a[WaitLink]
before AliasLink - Catalog[AliasLink] in
auth.reindex.cas
,live.reindex.cas
connectors. - Renamed CopyLink - Inventory (Copy To
Product)[CopyLink] to CopyLink - Inventory
(Copy To Product)[InventoryCopyLink] in
auth.inventory
,auth.reindex
,live.reindex
,push.to.live.cas
,push.to.live
connectors. - Added CacheLink[CacheLink] to refresh
controller service caches before continuing in
auth.reindex
,live.reindex
connectors. - Added product and category cache invalidation to dataload
connectors. Added dataload support to
InvalidateProduct
andInvalidateCategory
.
- Added WaitLink - Product Stage 1a[WaitLink],
WaitLink - Enrich NLP[WaitLink], and moved
Enrich NLP[NLPStage1] after
WaitLink - Product Stage 1g[WaitLink] in
the
- SQL file location change
- In previous versions, custom SQL was kept in the same template schema in the NiFi registry as other customization code. To make it easier for you to customize the SQL by itself, this SQL has been moved out to separate files, such as DatabaseProductStage.sql, in the src/main/resources/sql directory. The previous ComposeDatabaseSQL processor has been replaced with a new processor, GenerateSQL, that is used to retrieve these files. A shell script is used to update the SQL in the comment section of every processor, making it easier for you to access it from anywhere.
9.1.14.1 release changes
- ElasticSearch Schema
- No Change.
- NiFi Pipeline
- Added WaitLink - Product Stage 1a[WaitLink],
WaitLink - Enrich NLP[WaitLink], and moved
Enrich NLP[NLPStage1] after WaitLink -
Product Stage 1g[WaitLink] in the
dataload.product.cas
connector.
9.1.14 release changes
- ElasticSearch Schema
-
- Tokenizer switch in custom_splitter
analyzer:
Implemented a switch in the tokenizer used by the custom_splitter analyzer from keyword to whitespace in cas_product_elastic_schema.json.
- New Split field addition in
search index mapping:
Introduced a new split field in the search index mapping of cas_product_elastic_schema.json with specific settings tailored for text analysis.
- Tokenizer switch in custom_splitter
analyzer:
- NiFi Pipeline
-
- Added WaitLink - Enrich NLP[WaitLink] before
Clear TI Delta
Table[DatabaseClearTIDeltaTable] and moved
Enrich NLP[NLPStage1] up before
WaitLink - Enrich NLP[WaitLink] in
dataload.product
connector. - Added WaitLink - Product Stage 1a[WaitLink] and
Clear TI Delta
Table[DatabaseClearTIDeltaTable] in
auth.inventory.cas
connector. - Added Clear TI Delta
Table[DatabaseClearTIDeltaTable] in
auth.reindex.cas
connector. - Added MoveLink - Store[MoveLink] and
AliasLink - Store[AliasLink] in
push-to-live
andpush-to-live.cas
connectors. - Moved ContinueLink - Flow Control[ContinueLink] up before Inventory Stage 1a (Parent Inventories)[DatabaseInventoryStage1a].
- Moved
auth.postindex/live.postindex
connector stages to the mainauth.reindex/live.reindex
connector, leavingauth.postindex/live.postindex
connector now used for customization only. - Added new ContinueLink - Flow Control[ContinueLink]
in
auth.attribute
andlive.attribute
connectors.
- Added WaitLink - Enrich NLP[WaitLink] before
Clear TI Delta
Table[DatabaseClearTIDeltaTable] and moved
Enrich NLP[NLPStage1] up before
WaitLink - Enrich NLP[WaitLink] in
9.1.13.3 release changes
- ElasticSearch Schema
-
- Catalog filter enhancement:
Added support for base contract inheritance for catalog filters by introducing a new child field in store_dynamic_mappings.json.
- Default stemmer change to Snowball:
Changed the default stemmer in cas_product_elastic_schema.json and product_elastic_schema.json from the existing one to the Snowball stemmer.
- Catalog filter enhancement:
- NiFi pipeline
-
- Added WaitLink - Product Stage 1a[WaitLink] and
Clear TI Delta
Table[DatabaseClearTIDeltaTable] at end of
auth.inventory.cas
connector. - Added Clear TI Delta
Table[DatabaseClearTIDeltaTable] at the end of the
auth.reindex.cas
connector.
- Added WaitLink - Product Stage 1a[WaitLink] and
Clear TI Delta
Table[DatabaseClearTIDeltaTable] at end of
9.1.13.2 release changes
- ElasticSearch Schema
- No changes.
- NiFi pipeline
-
- Added WaitLink - Category Stage 1c[WaitLink]
and WaitLink - Product Stage 1g[WaitLink] to
the
dataload.category.cas
connector. Relocated Clear TI Delta Table[DatabaseClearTIDeltaTable] to the end of the pipeline - Added Clear TI Delta
Table[DatabaseClearTIDeltaTable] after ingest
completion in auth.inventory,
auth.reindex
,dataload.category.cas
,dataload.category
,dataload.product.cas
,dataload.product
,live.inventory
,live.reindex
connectors. - Changed NRTLink[CASNRTLink] to
DataloadLink[CASDataloadLink] in
auth.inventory.cas
,live.inventory.cas
connectors.
- Added WaitLink - Category Stage 1c[WaitLink]
and WaitLink - Product Stage 1g[WaitLink] to
the
9.1.13.1 release changes
- ElasticSearch Schema
-
- New field addition:
Added a new field named key inside attribute.value in both cas_product_elastic_schema.json and product_elastic_schema.json.
- New field addition:
9.1.13.0 release changes
- ElasticSearch Schema
-
- Natural keyword field mapping:
Modified the cas_product_elastic_schema.json and product_elastic_schema.json to include natural keyword field mapping.
- Asset store schema addition:
Introduced the asset_store to the store_dynamic_mappings.json.
- Text property details addition:
Added text property details for default.correction in product_elastic_schema.json.
- Parent field inclusion:
Under the name and description fields in product_elastic_schema.json, a new parent field has been added.
- Natural keyword field mapping:
- NiFi pipeline
-
- Removed Product Stage 1c (Find SEO
URLs)[DatabaseCASProductStage1c] from
live.reindex.cas
,live.product.cas
,dataload.product.cas
,auth.reindex.cas
, andauth.product.cas
connectors. - Combined CAS post index into the reindex process in
auth.reindex.cas
andlive.reindex.cas
connectors. - Removed Category Stage 1e (Find SEO
URL)[DatabaseCASCategoryStage1e] from
auth.category.cas
,auth.reindex.cas
,dataload.category.cas
,live.category.cas
, andlive.reindex.cas
connectors. - Added WaitLink - Product Stage 1g[WaitLink] in
the
auth.reindex.cas
andlive.reindex.cas
connectors. - Moved Store[DatabaseCASStoreStage1] stage
before the Product Schema[CASProductSchema]
stage in the
auth.reindex.cas
andlive.reindex.cas
connectors. - Added
copy.condition.expression
to all Auth and Live CopyLink for Price and Inventory. - Removed
auth.postindex.cas
andlive.postindex.cas
connectors. - Added Remove Workspace Information -
Attribute[RemoveWorkspaceDocument], Remove
Workspace Information -
Category[RemoveWorkspaceDocument], Remove
Workspace Information -
Product[RemoveWorkspaceDocument], Remove
Workspace Information - URL[RemoveWorkspaceDocument]
and Remove Workspace Information -
Description[RemoveWorkspaceDocument] stages in the
push-to-live
andpush-to-live.cas
connectors. - Added
auth.alias
andlive.alias
connectors to support switching catalog index aliases using API. - Changed SplitLink - Inventory[SplitLink] to
CopyLink - Inventory (Copy To
Product)[CopyLink] in the
push-to-live
connector. - Added
update-live
andupdate-live.cas
connectors. - Removed Category Stage 1d (Build
Hierarchy)[DatabaseCASCategoryStage1d] from the
auth.product.cas
andlive.product.cas
connectors. - Removed CopyLink - Price (Copy To
Product)[CopyLink], WaitLink - Price
CopyLink[WaitLink], CopyLink - Inventory
(Copy To Product)[CopyLink], and WaitLink -
Inventory CopyLink[WaitLink] from the
update-live.cas
andupdate-live connectors
. - Added WaitLink - Inventory CopyLink[WaitLink]
in
auth.inventory
andlive.inventory
connectors. - Removed LockLink - Store[LockLink], LockLink -
Catalog[LockLink], LockLink -
Attribute[LockLink], LockLink -
Category[LockLink], LockLink -
Product[LockLink], LockLink -
Page[LockLink], LockLink -
Description[LockLink], (Un)LockLink -
Store[LockLink], (Un)LockLink -
Catalog[LockLink], (Un)LockLink -
Attribute[LockLink], (un)LockLink -
Category[LockLink], (Un)LockLink -
Product[LockLink], (Un)LockLink -
Page[LockLink], and (Un)LockLink -
Description[LockLink] from the
push-to-live
andpush-to-live.cas
connectors. - Added Clear TI Delta
Table[DatabaseClearTIDeltaTable] after ingest
completion in
auth.inventory
auth.reindex
,dataload.category.cas
,dataload.category
,dataload.product.cas
,dataload.product
,live.inventory
, andlive.reindex
connectors. - Changed NRTLink[CASNRTLink] to
DataloadLink[CASDataloadLink] in the
auth.inventory.cas
andlive.inventory.cas
connectors.
- Removed Product Stage 1c (Find SEO
URLs)[DatabaseCASProductStage1c] from
9.1.12 release changes
- ElasticSearch Schema
- The following changes apply to category, product and attribute schemas.
- New analyzers
- These analyzers use different tokenizers and
filters, which are applied to different index fields to perform
text analysis processing on terms during ingestion and at index
time, and during query processing and search runtime.
- custom_splitter
- Uses keyword tokenizer.
- custom_shingle
- Uses standard tokenizer.
- custom_analyzer
- Uses standard tokenizer.
- New filters
- The following Elasticsearch filters can now be used inside
custom analyzers to convert (normalize, pre-process, etc.) the
index field values for use in the Elasticsearch index
- html_strip_filter
Strips html elements from text and replaces html entities with their decoded values.
- asciifolding
Converts alphabetic, numeric, and symbolic characters that are not in the Basic Latin Unicode block.
- word_delimiter_filter
Splits tokens at non-alphanumeric characters.
- shingle_filter
Adds shingles, or word n-grams, to a token stream by concatenating adjacent tokens.
- trim_filter
Removes leading and trailing whitespace from each token in a stream.
- html_strip_filter
- Normalizer property changed
- The normalizer property has changed from lowercase to normalized, because it not only changes the word to lowercase but also performs ASCII folding. This normalizer is used in all normalized index fields.
9.1.9.0 release changes
- ElasticSearch Schema
- No changes.
- NiFi pipeline
-
- URLStage2a and
URLStage2b have been removed along with
their Waitlinks.
URL Stage 1a has been updated to write to both URL Index and Category Index (add SEO) at the same time. This avoids issues where URL Stage 2a queries Elasticsearch prior to completion of the URL document.
Since URL Stage 1a (copy to category) already adds SEO to the Category Index, URL Stage 2a is no longer needed and has been removed. The same applies to URL Stage 1b (copy to product) and it has been removed.
- Removed CategoryStage4 (Category Deletion)
in NRT and Live pipelines.
CategoryStage4 (Category Deletion) has been removed from before Category Stage 1a (Main Document). This stage was previously used to handle edge cases for deletion. Since the catalog hierarchy now builds directly from the database, this stage is no longer needed and can be removed. This change improves overall indexing time.
- URLStage2a and
URLStage2b have been removed along with
their Waitlinks.
9.1.8.0 release changes
- ElasticSearch Schema
- No changes.
- NiFi pipeline
-
- Removed ProductStage 1f.
The rolling up (1e) and pushing down (1f) stages of the product attributes connector have been combined into one more performant stage, Product Stage 1e.
- CategoryStage2 has been changed to
Category Stage 1d and
redesigned.
Previously, CategoryStage2 built the category hierarchy by querying Elasticsearch. In Version 9.1.8, the category hierarchy is built by reading from the database for improved performance and reliability.
- Removed ProductStage 1f.
9.1.7.0 release changes
- ElasticSearch Schema
- No changes.
- NiFi pipeline
-
- The Find Child Items process group has been removed from
Product Stage 1b.
The
Find Child Items
process group is removed from Product Stage 1b. The logic of finding child items is moved to a new stage called Product Stage 1h. - Product Stage 1f (Rollup attributes) and
its
WaitLink
andRefreshLink
have been removed.The rollup logic has been combined into Product Stage 1e for better performance.
- Product Stage 2 (pushdown) and its
WaitLink
andRefreshLink
have been removed.The pushdown attribute logic has now been combined into Product Stage 1e for better performance. Other pushdown logic areas have been combined with other stages.
- Product Stage 1c (override child
attributes) and its
WaitLink
andRefreshLink
have been removed.The override child attribute logic is no longer needed with the new logic of Product Stage 1e
- Product Stage 1d (find item attachment) and
its
WaitLink
andRefreshLink
have been removed.The find item attachment logic is no longer needed with the new logic in Product Stage 1b.
- NRT/Dataload ProductStage4,
DatabaseProductStage1e-2 and
DatabaseProductStage1f-2 have been
removed.
The logic of these stages are no longer needed with the new optimized logic. ProductStage4 and ProductStage1e-2 have also been removed from the data load pipeline.
- The Find Child Items process group has been removed from
Product Stage 1b.