Minimal and recommended configuration tunable parameter values

A set of optimal processing values for the minimal configuration for NiFi and Elasticsearch are provided. These are the default values for the HCL Commerce deployment. They are sufficient for a typical index up to 300,000 catalog items. The second configuration provides optimal processing of the recommended configuration for NiFi and Elasticsearch. These are the provided recommended values for the HCL Commerce deployment. They are sufficient for a typical index up to 1M catalog items.

Heap sizes

Optimal values:

NiFi heap – 9GB
Elasticsearch heap - 12GB

Recommended values:

NiFi heap – 12GB 
Elasticsearch heap - 16GB

Processor groups threading changes

Nifi Flow>>Logging Service

Optimal values:

Nifi Flow>>Logging Service>>Generate Log Message: Configuration/Concurrent Tasks = 4 
Nifi Flow>>Logging Service>>Batch Bulk Document – Log: Configuration/Concurrent Tasks = 4

Recommended values:

Nifi Flow>>Logging Service>>Generate Log Message: Configuration/Concurrent Tasks = 8 
Nifi Flow>>Logging Service>>Batch Bulk Document – Log: Configuration/Concurrent Tasks = 8
All other processors are set at Concurrent Tasks= 1

All other processors are set at Concurrent Tasks = 1

Nifi Flow>>Bulk Service – Product

Optimal values:

Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Post Bulk Elasticsearch-Product: Configuration/Concurrent Tasks = 16`
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Set stage.end.time : Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Analyze Bulk Response: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search> Post Refresh Elasticsearch: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4

Recommended values:

Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Post Bulk Elasticsearch-Product: Configuration/Concurrent Tasks = 32
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Set stage.end.time : Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Analyze Bulk Response: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search> Post Refresh Elasticsearch: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8

Nifi Flow>>Bulk Service – Price

Optimal values:

Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Post Bulk Elasticsearch-Price: Configuration/Concurrent Tasks = 16
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4

Recommended values:

Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Post Bulk Elasticsearch-Price: Configuration/Concurrent Tasks = 32
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8

Nifi Flow>>Bulk Service – Inventory

Optimal values:

Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Post Bulk Elasticsearch-Inventory: Configuration/Concurrent Tasks = 16
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4

Recommended values:

Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Post Bulk Elasticsearch-Inventory: Configuration/Concurrent Tasks = 32
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8

Nifi Flow>>NLP Service

Optimal values:

Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>Generate Query: Configuration/Concurrent Tasks = 6
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>GET Elasticsearch: Configuration/Concurrent Tasks = 6
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>> SCROLL Elasticsearch: Configuration/Concurrent Tasks = 6
Nifi Flow>>NLP Service>>Parse NLP Messages>>NLP Processor>>Transform Document-CoreNLP Ingest Processor: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Track Bulk Request: Configuration/Concurrent Tasks = 4
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Post Bulk Elasticsearch: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Analyze Bulk Response: Configuration/Concurrent Tasks = 4
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Retry Flow File: Configuration/Concurrent Tasks = 4

Recommended values:

Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>Generate Query: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>GET Elasticsearch: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>> SCROLL Elasticsearch: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>NLP Processor>>Transform Document-CoreNLP Ingest Processor: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Track Bulk Request: Configuration/Concurrent Tasks = 8
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Post Bulk Elasticsearch: Configuration/Concurrent Tasks = 32
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Analyze Bulk Response: Configuration/Concurrent Tasks = 8
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Retry Flow File: Configuration/Concurrent Tasks = 8

Bucket size changes

1i

Optimal:

Nifi Flow> live reindex- DatabaseProductStage1i: Variables:scroll.bucket.size=5000

Recommended:

Nifi Flow> live reindex- DatabaseProductStage1i: Variables:scroll.bucket.size=5000

1e

Optimal:

Nifi Flow> live reindex- DatabaseProductStage1e: Variables:scroll.bucket.size=5000

Recommended:

Nifi Flow> live reindex- DatabaseProductStage1e: Variables:scroll.bucket.size=5000

Refresh rate change

NiFi Flow> live reindex- StoreSchema >> Setup Elasticsearch Index Schema : Populate Store Index Schema

Note: To change this value, stop the processor, click and edit the json object, and replace the refresh_interval value.