Skip to content

kube-prometheus-stack setup

Provides Prometheus (metrics) and Grafana (dashboards) for the cluster, including the Domino Workspace Grafana dashboards that ship with the Domino Helm chart.

Requires cert-manager if Grafana is exposed over HTTPS through APISIX.

Install

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Create kube-prometheus-stack-values.yaml with the following content:

crds:
  enabled: true

prometheus:
  prometheusSpec:
    # Discover ServiceMonitors from every chart, not just this Helm release's own
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}

helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace kube-prometheus-stack \
  --create-namespace \
  -f kube-prometheus-stack-values.yaml

serviceMonitorSelectorNilUsesHelmValues=false is required so Prometheus picks up ServiceMonitors created by other charts (such as Domino's, which uses the release label kube-prometheus-stack by default — see serviceMonitor.labels in HCL Domino) instead of only ones from this Helm release.

Optional: sign in to Grafana with Domino's OIDC provider

Once HCL Domino is deployed, it registers a grafana OIDC client (oidc.clients.grafana). To use it for Grafana login, add to kube-prometheus-stack-values.yaml and re-run the helm upgrade:

grafana:
  grafana.ini:
    server:
      root_url: https://grafana.<domain>
    auth.generic_oauth:
      enabled: true
      client_id: grafana
      client_secret: <client secret configured in Domino's oidc.clients.grafana>
      scopes: openid email profile
      auth_url: https://auth.<domain>/auth/protocol/oidc/auth
      token_url: https://auth.<domain>/auth/protocol/oidc/token
      api_url: https://auth.<domain>/auth/protocol/oidc/userinfo

Optional: expose Grafana through APISIX

Add to kube-prometheus-stack-values.yaml (merge under the existing grafana: key if OIDC is also configured above) and re-run the helm upgrade:

grafana:
  ingress:
    enabled: true
    ingressClassName: apisix
    hosts:
      - grafana.<domain>

Notes

  • If running on RKE2, kube-proxy's Service uses component/tier labels instead of the standard k8s-app label — without kubeProxy.service.selector: {component: kube-proxy, tier: control-plane} its ServiceMonitor finds no pods. Also add prometheus-node-exporter.tolerations for the CriticalAddonsOnly taint RKE2 applies to control-plane nodes, so node-exporter also runs there — same toleration shown in Alloy setup.
  • Persistent storage for Prometheus is configured under prometheus.prometheusSpec.storageSpec — set a StorageClass and size appropriate for the retention period needed.

See the chart's own documentation for the full set of options, and dw_requirements.md for sizing guidance.

Verify

kubectl get pods -n kube-prometheus-stack