Configuring logging for Kubernetes with Elastic, Fluentd, and Kibana

You can configure centralized logging for your Kubernetes deployment using Elastic, Fluentd, and Kibana (EFK).

Using this selection of tools:
  • Fluentd collects logs from pods running in the cluster and forwards them to Elastic.
  • Elastic stores, indexes, and searches logs.
  • Kibana connects to Elastic to generate queries and to visualize results.

Use this document as a starting point for your centralized logging solution. You can use other software, products, or guides to further customize your logging solution to suite your business requirements.

Before you begin

  • Ensure that you have a Kubernetes cluster with Helm installed.
  • Ensure that kubectl command line (kubectl CLI) is installed.
  • Obtain the sample Helm Chart deployment configuration files within the sample_values directory of your cloned Commerce+ Helm Chart Git project.
  • Depending on your deployment type, enable JSON logging for your Commerce+ services by using the common.enableJsonLogging Vault configuration or Docker container configuration variables.

Procedure

  1. Create a namespace for your EFK stack. 
    kubectl create namespace log
  2. Add the bitnami Helm repository. 
    helm repo add bitnami https://charts.bitnami.com/bitnami
  3. Deploy an instance of Elastic for logging usage.
    1. Deploy Elastic using the sample log-elasticsearch-values.yaml configuration file from the sample_values directory of your cloned Commerce+ Helm Chart Git project.
      helm install elasticsearch bitnami/elasticsearch -n log -f log-elasticsearch-values.yaml --version "15.3.0"
    2. Monitor the deployment and ensure that all pods are healthy.
  4. Deploy Kibana.
    1. Deploy Kibana using the sample log-kibana-values.yaml configuration file from the sample_values directory of your cloned Commerce+ Helm Chart Git project.
      helm install kibana bitnami/kibana -n log -f log-kibana-values.yaml --version "8.1.3"
    2. Monitor the deployment and ensure that all pods are healthy.
  5. Configure and deploy Fluentd.
    1. Create a configuration map with the sample log-forwarder-configmap.yaml configuration map.
      kubectl apply -f log-forwarder-configmap.yaml -n log
      Note: If you use a namespace other than log you must update the value used in this sample file.
      Change "log" in the line below to the namespace that was created in step #1.
      host fluentd-0.fluentd-headless.log.svc.cluster.local
    2. Deploy Fluentd using the sample log-fluentd-values.yaml configuration file from the sample_values directory of your cloned Commerce+ Helm Chart Git project.
      helm install fluentd bitnami/fluentd -n log -f log-fluentd-values.yaml --version "3.7.4"
    3. Monitor the deployment and ensure that all pods are healthy.

Results

A centralized logging solution is implemented for your Commerce+ services running on Kubernetes.