HCL Commerce Version 9.1.15.0 or later

Metrics for tuning NiFi

Factors to consider when calculating tuning parameters for each stage and the metrics to collect from each stage. The importance of monitoring resource utilization is emphasized.

About this task

When calculating tuning parameters for each stage, ensure that you understand the metrics that you are collecting from each stage, and remember to monitor resource use at each stage. Factors to consider include Elasticsearch CPU, database CPU, NiFi CPU and heap memory, as well as catalog data size. Metrics such as page size, bucket size, and thread counts should be collected from each stage after each run. You can store this data in its own Elasticsearch index. Additional metrics to collect include elapsed time, amount of data processed, and resource use for all underlying capacities.

Procedure

  1. Create an Elasticsearch index to store the data you collect for later analysis.
  2. Consider the following factors when calculating your metrics to define a formula for the tuning parameters of each stage.
    • Elasticsearch CPU and heap memory
    • Database CPU
    • NiFi CPU and heap memory
    • Catalog data size.
    • SQL fetch size
  3. Metrics to be collected from each stage after each run and stored in Elasticsearch:
    • Page size, bucket size, thread counts
    • Elapsed time and amount of data processed
    • Resource utilization (CPU, heap, cache) on all underlying capacities