Skip to content

Alloy setup

Grafana Alloy ships node and pod logs, plus Kubernetes events, to Loki. This is separate from the optional per-pod Alloy sidecar that the Domino Helm chart can run for notes.log/Traveler logs (alloy.enabled) — this is a cluster-wide DaemonSet covering everything else.

Requires Loki to be reachable, and the cluster CA ConfigMap from trust-manager if Loki's gateway uses a certificate issued by the internal CA.

Install

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Create alloy-values.yaml with the following content:

alloy:
  mounts:
    varlog: true
    extra:
      - name: ca-bundle
        mountPath: /etc/alloy-ca
        readOnly: true
  configMap:
    content: |
      loki.write "loki" {
        endpoint {
          url = "https://loki-gateway.loki:443/loki/api/v1/push"
          tls_config {
            ca_file = "/etc/alloy-ca/ca.crt"
          }
        }
      }

      local.file_match "node_logs" {
        path_targets = [{
            __path__  = "/var/log/messages",
            job       = "node/messages",
            node_name = sys.env("HOSTNAME"),
            cluster   = "<cluster-name>",
        }]
      }

      loki.source.file "node_logs" {
        targets    = local.file_match.node_logs.targets
        forward_to = [loki.write.loki.receiver]
      }

      discovery.kubernetes "pod" {
        role = "pod"
        selectors {
          role  = "pod"
          field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
        }
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pod.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          action        = "replace"
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          action        = "replace"
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          action        = "replace"
          target_label  = "container"
        }
      }

      loki.source.kubernetes "pod_logs" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [loki.process.pod_logs.receiver]
      }

      loki.process "pod_logs" {
        stage.static_labels {
          values = { cluster = "<cluster-name>" }
        }
        forward_to = [loki.write.loki.receiver]
      }

      loki.source.kubernetes_events "cluster_events" {
        job_name   = "integrations/kubernetes/eventhandler"
        log_format = "logfmt"
        forward_to = [loki.process.cluster_events.receiver]
      }

      loki.process "cluster_events" {
        forward_to = [loki.write.loki.receiver]
        stage.static_labels {
          values = { cluster = "<cluster-name>" }
        }
      }
  enableReporting: false

controller:
  volumes:
    extra:
      - name: ca-bundle
        configMap:
          name: cluster-ca-bundle

helm upgrade --install alloy grafana/alloy \
  --namespace alloy \
  --create-namespace \
  -f alloy-values.yaml

If running on RKE2, also add a toleration for the CriticalAddonsOnly taint RKE2 applies to control-plane nodes, so the DaemonSet also collects logs from them — add to alloy-values.yaml under the existing controller: key, and re-run the helm upgrade:

controller:
  tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Equal"
      value: "true"
      effect: "NoExecute"

See the official installation guide for other configuration options.

Verify

kubectl get pods -n alloy