Prometheus Setup

Procedure

Install Prometheus using Helm.
Configure Prometheus to scrape metrics endpoints. Each product exposes metrics using Prometheus exporter JAR. Prometheus needs to scrape those metrics.
Validate data collection by querying sample metrics (CPU, heap usage, threads).

Installing Prometheus

Procedure

Add the Prometheus Helm repository and update it using the following commands:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install Prometheus using the following command:

helm install hcl-prometheus prometheus-community/prometheus

Modify values.yaml to enable remote write receiver:
```
server:
     extraFlags:
       - --web.enable-remote-write-receiver
```

Ensure that the scrape configurations are proper by modifying the ConfigMap (prometheus scrape-config.txt). A sample scrape config file is as follows:

 - job_name: 'otel-collector'
      scrape_interval: 10s
      static_configs:
        - targets: ['otel-collector.monitoring.svc.cluster.local:8888']
    # Scrape configuration for unica-platform
    - job_name: 'unica-platform'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        # Keep only pods labeled with `app_kubernetes_io_name=unica-platform`
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
          action: keep
          regex: unica-platform
        # Keep only pods labeled with `app_kubernetes_io_instance=hcl`
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
          action: keep
          regex: hcl
        # Keep only pods in Running state
        - source_labels: [__meta_kubernetes_pod_phase]
          action: keep
          regex: Running
        # Replace the pod IP with JMX agent port (9124)
        - source_labels: [__meta_kubernetes_pod_ip]
          target_label: __address__
          replacement: $1:9124
        # Add pod name to metadata
        - source_labels: [__meta_kubernetes_pod_name]
          target_label: pod
        # Add namespace as a target label
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        # Add custom job label to differentiate it
        - target_label: job
          replacement: unica-platform
        # Add the instance as a target label
        - source_labels: [__address__]
          target_label: instance
        - target_label: application
          replacement: unica-platform  # Add this label
    # Scrape configuration for unica-journeyweb
    - job_name: 'unica-journeyweb'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        # Keep only pods labeled with `app_kubernetes_io_name=unica-journeyweb`
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
          action: keep
          regex: unica-journeyweb
        # Keep only pods labeled with `app_kubernetes_io_instance=hcl`
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
          action: keep
          regex: hcl
        # Keep only pods in Running state
        - source_labels: [__meta_kubernetes_pod_phase]
          action: keep
          regex: Running
        # Replace the pod IP with JMX agent port (9137)
        - source_labels: [__meta_kubernetes_pod_ip]
          target_label: __address__
          replacement: $1:9137
        # Add pod name to metadata
        - source_labels: [__meta_kubernetes_pod_name]
          target_label: pod
        # Add namespace as a target label
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        # Add custom job label to differentiate it
        - target_label: job
          replacement: unica-journeyweb
        # Add the instance as a target label
        - source_labels: [__address__]
          target_label: instance
        - target_label: application
          replacement: unica-journeyweb  # Add this label
    # Scrape configuration for unica-campaign
    - job_name: 'unica-campaign'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        # Keep only pods labeled with `app_kubernetes_io_name=unica-campaign`
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
          action: keep
          regex: unica-campaign
        # Keep only pods labeled with `app_kubernetes_io_instance=hcl`
        - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
          action: keep
          regex: hcl
        # Keep only pods in Running state
        - source_labels: [__meta_kubernetes_pod_phase]
          action: keep
          regex: Running
        # Replace the pod IP with JMX agent port (9124)
        - source_labels: [__meta_kubernetes_pod_ip]
          target_label: __address__
          replacement: $1:9124
        # Add pod name to metadata
        - source_labels: [__meta_kubernetes_pod_name]
          target_label: pod
        # Add namespace as a target label
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        # Add custom job label to differentiate it
        - target_label: job
          replacement: unica-campaign
        # Add the instance as a target label
        - source_labels: [__address__]
          target_label: instance
        - target_label: application
          replacement: unica-campaign  # Add this label
    - job_name: 'nginx-ingress'
      static_configs:
        - targets: ['comp-4976-1.nonprod.hclpnp.com:10254']
      metrics_path: '/metrics'
      scheme: 'http'
    - job_name: 'unica interact'
      static_configs:
        - targets: ['10.115.147.43:9011']
      metrics_path: '/metrics'
      scheme: 'http'
      relabel_configs:
        - target_label: application
          replacement: unica-interact  # Add this label
    - job_name: prometheus
      static_configs:
      - targets:
        - localhost:9090
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token