Configuring logging for Kubernetes with Elasticsearch, Fluentd, and Kibana
You can configure centralized logging for your Kubernetes deployment using Elasticsearch, Fluentd, and Kibana (EFK).
Using this selection of tools:- Fluentd collects logs from pods running in the cluster and forwards them to Elasticsearch.
- Elasticsearch stores, indexes, and searches logs.
- Kibana connects to Elasticsearch to generate queries and to visualize results.
Use this document as a starting point for your centralized logging solution. You can use other software, products, or guides to further customize your logging solution to suite your business requirements.
Before you begin
- Ensure that you have a Kubernetes cluster with Helm installed.
- Ensure that kubectl command line (kubectl CLI) is installed.
- Obtain the sample Helm Chart deployment configuration files within the sample_values directory of your cloned HCL Commerce Helm Chart Git project.
- Depending on your deployment type, enable JSON logging for your
HCL Commerce services by using the
common.enableJsonLogging Vault configuration or Docker container
configuration variables. For more information, see:
Procedure
-
Create a namespace for your EFK stack.
kubectl create namespace log
-
Add the bitnami Helm repository.
helm repo add bitnami https://charts.bitnami.com/bitnami
-
Deploy an instance of Elasticsearch for logging usage.
- Deploy Elasticsearch using the sample
log-elasticsearch-values.yaml configuration file from the
sample_values directory of your cloned HCL Commerce
Helm Chart Git
project.
helm install elasticsearch bitnami/elasticsearch -n log -f log-elasticsearch-values.yaml --version "15.3.0"
- Monitor the deployment and ensure that all pods are healthy.
- Deploy Elasticsearch using the sample
log-elasticsearch-values.yaml configuration file from the
sample_values directory of your cloned HCL Commerce
Helm Chart Git
project.
-
Deploy Kibana.
- Deploy Kibana using the sample log-kibana-values.yaml
configuration file from the sample_values directory of your
cloned HCL Commerce Helm Chart Git
project.
helm install kibana bitnami/kibana -n log -f log-kibana-values.yaml --version "8.1.3"
- Monitor the deployment and ensure that all pods are healthy.
- Deploy Kibana using the sample log-kibana-values.yaml
configuration file from the sample_values directory of your
cloned HCL Commerce Helm Chart Git
project.
-
Configure and deploy Fluentd.
- Create a configuration map with the sample
log-forwarder-configmap.yaml configuration
map.
kubectl apply -f log-forwarder-configmap.yaml -n log
Note: If you use a namespace other thanlog
you must update the value used in this sample file.Change "log" in the line below to the namespace that was created in step #1.host fluentd-0.fluentd-headless.log.svc.cluster.local
- Deploy Fluentd using the sample log-fluentd-values.yaml
configuration file from the sample_values directory of your
cloned HCL Commerce Helm Chart Git
project.
helm install fluentd bitnami/fluentd -n log -f log-fluentd-values.yaml --version "3.7.4"
- Monitor the deployment and ensure that all pods are healthy.
- Create a configuration map with the sample
log-forwarder-configmap.yaml configuration
map.