Multiple or Concurent Elasticsearch operations

Detecting and determining the root cause of Elasticsearch slowness when concurrent operations are executed is complex and should be performed with extra care and preparation. You can anticipate carrying out environment testing in the following situations:

As anticipated for the peak season, when the production workload is at capacity or at its peak.
All the index-related operations that are expected to be executed during the peak workload (index build, NRT updates to the index, inventory index updates, etc.).
Other operations do not directly affect the index or search data but can affect the overall operations (for example, full cache clear after reindexing, etc.).

The case of flow congestion

Certain situations are frequently encountered when executing a mixed workload, such as funnel congestion and performance degradation. Differing from the single type of operation, this is a more complicated tuning exercise that observes and adjusts for insufficient resources and balances those resources according to some priority. For example, the production search request should precede the index build operations.

In a case where the slowness of Elasticsearch is due to a workload mix in which the search cluster was indexing and serving live request data, the tuning comes more complicated and requires upfront decision-making regarding the priorities of the workload mix types.

In most cases, you need to prioritize one type of operation over another. In this example, it would require selecting a higher priority for the Live searches and a lower priority for the index creation on the Auth environment.

Changing write threads:

Changing the write and search threads in Elasticsearch can effectively tune the cluster for better performance and provide prioritization between the write (Auth index build) and Live (search threads requests) operations.

Elasticsearch has a write thread pool that handles indexing requests. By default, this pool has a size of (number of CPU cores * 2) + 1. You can increase or decrease the size of this pool based on the workload of your cluster.

To change the size of the write thread pool, you can use the thread_pool.write.size setting in the Elasticsearch configuration file. For example, to increase the size of the write thread pool to 16, you can add the following line to the configuration file:

thread_pool.write.size : 16.

Changing search threads

Elasticsearch also has a search thread pool that handles search requests. By default, this pool has a size of (number of CPU cores * 3) / 2. You can increase or decrease the size of this pool based on the workload of your cluster.

To change the size of the search thread pool, you can use the thread_pool.search.size setting in the Elasticsearch configuration file. For example, to increase the size of the search thread pool to 24, you can add the following line to the configuration file:

thread_pool.search.size: 24.

Important: Changing the thread pool sizes should be done cautiously and after thorough testing. Increasing the thread pool size too much can lead to resource exhaustion and cause performance issues. It is recommended that you monitor the resource usage of your cluster after changing the thread pool sizes and adjust them accordingly to achieve optimal performance.

Adjusting index settings can also effectively tune Elasticsearch for better performance. The following parameters can be changed to enhance indexing performance:

Refresh interval

By default, Elasticsearch refreshes the search index every second. This means that when a document is indexed, it will be immediately available for search at the next refresh interval. If you have a high volume of indexing requests, consider increasing the refresh interval to reduce the frequency of index refreshes and improve indexing performance. Conversely, if you need real-time indexing, you can decrease the refresh interval.

To adjust the refresh interval, use the index.refresh_interval setting. For example, to set the refresh interval to 5 seconds, you can run the following command:

PUT /my_index/_settings
{
  "index" : {
    "refresh_interval" : "5s"
  }
}

Number of shards

The number of shards in an index can also affect indexing performance. If you have an extensive index with many shards, indexing performance may be slower due to the overhead of coordinating writes across multiple shards. In general, keeping the number of shards per index between 1 and 5 is recommended.

To adjust the number of shards, you can use the index.number_of_shards setting. For example, to set the number of shards to 3, you can run the following command:

PUT /my_index/_settings
{
  "index" : {
    "number_of_shards" : 3
  }
}