ELK Overview
The ELK Stack is a collection of three open-source products: Elasticsearch, Logstash, and Kibana.
The features of the ELK stack are as follows:
- Centralized logging and monitoring to identify problems with containers, or applications, hosted inside the pod
- Visualization tools to represent data using a graph or a chart
- Host ELK as a standalone system or a Docker container or a Kubernetes pod
Filebeat
Filebeat collects and ships log files, and is also the most commonly used beat. You can install Filebeat on almost any operating system, or also as a Docker container. It includes internal modules for platforms like Apache, MySQL, Docker, MariaDB, Kafka, and many more.
Filebeat is very efficient and it displays this in how it handles backpressure. If Logstash is busy, Filebeat slows down its read rate and picks up the beat once the slowdown is completed.
For Unica, Filebeat is deployed as a sidecar container inside Platform pod, and Platform and Filebeat, as a sidecar, runs as a container inside a pod. The Filebeat has a read-only access to persistent volume. It reads the logs from the logs folder, specified in configmap for all Unica applications, to keep sending logs to ELK.
metricbeat
Metricbeat collects and reports various system-level metrics for various systems and platforms. Metricbeat also supports internal modules for collecting statistics from specific platforms.
For Unica, metricbeat is deployed as
a deamonset in kube-system
namespace to collect metrics from the
metric-server in OpenShift. In Kubernetes, it connects to both the Kubernetes
API-server and the metric server.
Fluentd
It is a smart metrics and log shipper. For Unica, Fluentd is deployed as a deamonset in custom namespace to collect metrics from Kubernetes.
Advantages of using ELK
Logging , keeping historical logs, or monitoring the logs is a real challenge in containerized applications. If you destroy a container, everything is lost, including the logs. The advantages of using ELK are as follows:
- Maintains and keeps all the data, and makes this activity easy, even if the cluster, pod, or node is destroyed.
- Allows searching of all the logs in a single place.
- Helps find issues in multiple servers, or pods, by connecting logs during a specific time frame.