Filtering out logs

Filter out Component Pack for HCL Connections™ logs so that they are excluded from data collection and are not used in the Kibana dashboard's analytics.

Using Helm, you can instruct Logstash to filter out logs from one or more Component Pack containers.

  • By default, all logs from Filebeat and Logstash are filtered out. You can additionally filter out logs from the Component Pack and system containers listed at the end of this topic.
  • To filter out additional logs, use the logstash.filterOut Helm parameter when installing Elastic Stack, and set it to the list of container names to filter out.

Syntax

When specifying the list of containers for filtering out logs, enclose the list in quotation marks, and use a blank space as the delimiter between names. For example:

  • logstash.filterOut="sanity"
  • logstash.filterOut="sanity redis-server"

Examples of Helm commands with the logstash.filterOut parameter

The following example demonstrates filtering out logs from the "sanity" container:

helm install \  
--name=elasticstack extractedFolder/microservices_connections/hybridcloud/helmbuilds/elasticstack-0.1.0-20191122-115449.tgz \  
--set \  
global.onPrem=true,\  
global.image.repository=Docker_registry/connections,\
logstash.filterOut="sanity"

You can also filter out logs from system pods. The following example demonstrates filtering out logs from the "sanity" and "redis-server" containers, plus the "calico-node" pod:

helm install \  
--name=elasticstack extractedFolder/microservices_connections/hybridcloud/helmbuilds/elasticstack-0.1.0-20191122-115449.tgz \  
--set \  
global.onPrem=true,\  
global.image.repository=Docker_registry/connections,\
logstash.filterOut="sanity redis-server calico-node"

Filtering out additional logs while Logstash is running

If Logstash is running, you don't have to shut it down to add containers to the logstash.filterOut list. Just perform a Helm upgrade and specify the additional container names you want to filter out. A rolling update will be performed where each pod is restarted with the new settings in place. The new list replaces the old list, so if you previously filtered out a container, be sure to include it in the new list.

The following example shows the original Helm command that was used to install Elastic Stack and filter out the "sanity" log, and the update command that modifies the logstash.filterOut list filter out the "sanity" log and the "redis-server" log.

The helm install command with the original logstash.filterOut list:

Prior state
helm install \  
--name=elasticstack extractedFolder/microservices_connections/hybridcloud/helmbuilds/elasticstack-0.1.0-20191122-115449.tgz \  
--set \  
global.onPrem=true,\  
global.image.repository=Docker_registry/connections,\
logstash.filterOut="sanity"

The helm update command with the revised logstash.filterOut list:

helm upgrade \  
elasticstack extractedFolder/microservices_connections/hybridcloud/helmbuilds/elasticstack-0.1.0-20191122-115449.tgz \  
--set \  
global.onPrem=true,\  
global.image.repository=Docker_registry/connections,\
logstash.filterOut="sanity redis-server"

You can also change the logstash.filterOut to stop filtering out logs, as shown in the following example where the logstash.filterOut is modified to remove the "sanity" log so that Sanity can be monitored.

The helm update command with the logstash.filterOut list revised to stop filtering out "sanity" logs:

helm upgrade \  
elasticstack extractedFolder/microservices_connections/hybridcloud/helmbuilds/elasticstack-0.1.0-20191122-115449.tgz \  
--set \  
global.onPrem=true,\  
global.image.repository=Docker_registry/connections,\
logstash.filterOut="redis-server"

Component Pack container names

Logs are created for the following containers. You can filter any Component Pack container's logs so that they are not included in the data that is stored in Elasticsearch for analysis in the Kibana dashboard.

  • analysisservice
  • appregistry-client
  • appregistry-service
  • bootstrap
  • community-suggestions
  • elasticsearch-curator
  • es-client
  • es-data
  • es-master
  • haproxy
  • indexingservice
  • itm-services
  • kibana
  • mail-service
  • middleware-graphql
  • mongo
  • mongo-sidecar
  • orient-web-client
  • people-idmapping
  • people-migrate
  • people-relation
  • people-scoring
  • redis-sentinel
  • redis-server
  • retrievalservice
  • sanity
  • sanity-watcher
  • solr
  • userprefs-service
  • zookeeper

System container names

Logs are created for the following system containers.

  • calico
  • calico-node
  • coredns
  • etcd
  • heapster
  • influxdb
  • kube-apiserver
  • kube-controller-manager
  • kube-proxy
  • kubernetes-dashboard
  • kube-scheduler
  • tiller