Recommended Configuration Tunable Parameter Values

For best performance results, you may want to change the default parameter values for Elastic and NiFi. A recommended configuration is presented here.

HEAP Sizes

NIFI Heap – 12GB 
ES Heap   - 16GB

Processor groups threading changes

Nifi Flow > Logging Service

Nifi Flow>>Logging Service>>Generate Log Message: Configuration/Concurrent Tasks = 8 
Nifi Flow>>Logging Service>>Batch Bulk Document – Log: Configuration/Concurrent Tasks = 8

All other processors are set at Concurrent Tasks= 1

Nifi Flow > Bulk Service – Product

Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Post Bulk Elastic-Product: Configuration/Concurrent Tasks = 32
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Set stage.end.time : Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Analyze Bulk Response: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search> Post Refresh Elastic: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8

Nifi Flow > Bulk Service – Price

Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Post Bulk Elastic-Price: Configuration/Concurrent Tasks = 32
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8

Nifi Flow > Bulk Service – Inventory

Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Post Bulk Elastic-Inventory: Configuration/Concurrent Tasks = 32
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 8
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 8

Nifi Flow > NLP Service

Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elastic>>Generate Query: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elastic>>GET Elastic: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elastic>> SCROLL Elastic: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>NLP Processor>>Transform Document-CoreNLP Ingest Processor: Configuration/Concurrent Tasks = 16

Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elastic>> Track Bulk Request: Configuration/Concurrent Tasks = 8
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elastic>> Post Bulk Elastic: Configuration/Concurrent Tasks = 32
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elastic>> Analyze Bulk Response: Configuration/Concurrent Tasks = 8
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elastic>> Retry Flow File: Configuration/Concurrent Tasks = 8

Bucket size changes

1i

Nifi Flow> live reindex- DatabaseProductStage1i: Variables:scroll.bucket.size=5000

1e
Nifi Flow> live reindex- DatabaseProductStage1e: Variables:scroll.bucket.size=5000

Refresh rate change

NIfi Flow> live reindex- StoreSchema >> Setup Elastic Index Schema : Populate Store Index Schema

Note: Stop the processor, click and edit the json object, replacing the refresh_intervalvalue with the desired value.