Minimal and recommended configuration tunable parameter values
Two configurationsof the NiFi tunable parameter values are presented, a minimal and an optimal value.
A set of optimal processing values for the minimal configuration for NiFi and Elasticsearch are provided. These are the default values for the HCL Commerce deployment. They are sufficient for a typical index up to 300,000 catalog items. The second configuration provides optimal processing of the recommended configuration for NiFi and Elasticsearch. These are the provided recommended values for the HCL Commerce deployment. They are sufficient for a typical index up to 1M catalog items.
- Heap sizes
- Optimal values:
Recommended values:NiFi heap – 9GB Elasticsearch heap - 12GB
NiFi heap – 12GB Elasticsearch heap - 16GB
Processor groups threading changes
- Nifi Flow>>Logging Service
- Optimal
values:
Recommended values:Nifi Flow>>Logging Service>>Generate Log Message: Configuration/Concurrent Tasks = 4 Nifi Flow>>Logging Service>>Batch Bulk Document – Log: Configuration/Concurrent Tasks = 4
Nifi Flow>>Logging Service>>Generate Log Message: Configuration/Concurrent Tasks = 8 Nifi Flow>>Logging Service>>Batch Bulk Document – Log: Configuration/Concurrent Tasks = 8 All other processors are set at Concurrent Tasks= 1
All other processors are set at Concurrent Tasks = 1
- Nifi Flow>>Bulk Service – Product
- Optimal
values:
Recommended values:Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Post Bulk Elasticsearch-Product: Configuration/Concurrent Tasks = 16` Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Set stage.end.time : Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Analyze Bulk Response: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search> Post Refresh Elasticsearch: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Post Bulk Elasticsearch-Product: Configuration/Concurrent Tasks = 32 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Set stage.end.time : Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Analyze Bulk Response: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search> Post Refresh Elasticsearch: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8
- Nifi Flow>>Bulk Service – Price
- Optimal
values:
Recommended values:Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Post Bulk Elasticsearch-Price: Configuration/Concurrent Tasks = 16 Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Post Bulk Elasticsearch-Price: Configuration/Concurrent Tasks = 32 Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8
- Nifi Flow>>Bulk Service – Inventory
- Optimal
values:
Recommended values:Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Post Bulk Elasticsearch-Inventory: Configuration/Concurrent Tasks = 16 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Post Bulk Elasticsearch-Inventory: Configuration/Concurrent Tasks = 32 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8 Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8
- Nifi Flow>>NLP Service
- Optimal
values:
Recommended values:Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>Generate Query: Configuration/Concurrent Tasks = 6 Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>GET Elasticsearch: Configuration/Concurrent Tasks = 6 Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>> SCROLL Elasticsearch: Configuration/Concurrent Tasks = 6 Nifi Flow>>NLP Service>>Parse NLP Messages>>NLP Processor>>Transform Document-CoreNLP Ingest Processor: Configuration/Concurrent Tasks = 16 Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Track Bulk Request: Configuration/Concurrent Tasks = 4 Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Post Bulk Elasticsearch: Configuration/Concurrent Tasks = 16 Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Analyze Bulk Response: Configuration/Concurrent Tasks = 4 Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Retry Flow File: Configuration/Concurrent Tasks = 4
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>Generate Query: Configuration/Concurrent Tasks = 16 Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>GET Elasticsearch: Configuration/Concurrent Tasks = 16 Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>> SCROLL Elasticsearch: Configuration/Concurrent Tasks = 16 Nifi Flow>>NLP Service>>Parse NLP Messages>>NLP Processor>>Transform Document-CoreNLP Ingest Processor: Configuration/Concurrent Tasks = 16 Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Track Bulk Request: Configuration/Concurrent Tasks = 8 Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Post Bulk Elasticsearch: Configuration/Concurrent Tasks = 32 Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Analyze Bulk Response: Configuration/Concurrent Tasks = 8 Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Retry Flow File: Configuration/Concurrent Tasks = 8
Bucket size changes
- 1i
- Optimal:
Recommended:Nifi Flow> live reindex- DatabaseProductStage1i: Variables:scroll.bucket.size=5000
Nifi Flow> live reindex- DatabaseProductStage1i: Variables:scroll.bucket.size=5000
- 1e
- Optimal:
Recommended:Nifi Flow> live reindex- DatabaseProductStage1e: Variables:scroll.bucket.size=5000
Nifi Flow> live reindex- DatabaseProductStage1e: Variables:scroll.bucket.size=5000
Refresh rate change
NiFi Flow> live reindex- StoreSchema >> Setup Elasticsearch Index Schema : Populate Store Index Schema