When and how to tune your Ingest pipelines
Tuning Ingest dataflow is necessary to avoid overloading or idling the system. An overloaded or underused system can experience performance issues.
When a large number of retries is observed during an ingest run, this is a typical sign of a bottleneck in an overloaded system. Conversely, the system could be under-utilized if idling is frequently observed in multiple stages of the Ingest data flow. Both of these conditions are indications that tuning is needed.
You should not expect to do performance tuning just once, during your deployment of HCL Commerce After operating for some time, certain components may no longer be at an optimal setting. In such a case you need to re-tune the system. The following scenarios are indicators that it is time to perform Ingest (NiFi) tuning.:
- At initial store or catalog deployment
- After a significant change in catalog data size
- After an (Ingest related) underlying capacity adjustment
- After a customization, software stack, or fixpack upgrade
- After encountering a resource constraint in Production due to Live traffic
Workflow
- Pick a connector
- Start tuning the heaviest Ingest connector first. This is the Auth and Live Re-indexing connector. This approach uses the largest possible dataflow as the baseline for tuning. Once this heaviest Ingest connector is tuned correctly for ingesting the current catalog data profile, the dataload and Near Real-Time (NR) connectors can use these tuning settings as their starting point.
- Pick a Pipeline
- When tuning the largest Ingest connector, start tuning the heaviest pipeline of this connector first, which is the product pipeline. Only move to the next pipeline when the current pipeline tuning is completed.
- Pick a Stage
- When tuning a pipeline, it is recommended that you start from the beginning and work your way through each stage down this pipeline.
- Tune each component in Extraction, Transformation, and Loading separately
- When tuning components within any given stage of a pipeline, do not mix several tuning changes and test them together. It is easier to observe the impact of a single change. Additionally, it is easier to begin tuning with the recommended settings and gradually increase the tuning parameters until the success criteria can no longer be met. This is a common approach used to determine an optimal configuration for your system.