Tuning of Elasticsearch Configuration - Parameter Optimization Through Simultaneous Perturbation Stochastic Approximation Algorithm

Abstract

By default, Elasticsearch configuration does not change while it receives data. However, when Elasticsearch stores a large amount of data over time, the default configuration becomes an obstacle in scaling for better performance. Besides, the machine that hosts Elasticsearch will have limitations on its specifications, like memory size. A solution to this problem is to tune the parameter configuration of Elasticsearch, which leads to achieving better performance. One way to tune parameters is by using Simultaneous Perturbation Stochastic Approximation. This report provides an implementation of optimizing Elasticsearch configuration parameters by observing the performance and automatically change the configuration to provide better performance. The used implementation relies on combining machine learning with ELasticsearch. Through this combination, Elasticsearch configuration can change its configuration parameters automatically without the need to reset the currently running instance of Elasticsearch. The results showed a good improvement in the number of inserted data and response time of the system.