Deploying CDP S3 Sink Connector

This section describes the steps to configure and deploy the HCL CDP S3 Sink Connector on an AKS cluster using the provided Helm chart.

Before you begin

The required HCL CDP Helm charts and container images must be downloaded and pushed to the container registry. For more information, see the Downloading HCL CDP Artifacts section.

Create the CDP S3 Sink Connector secret in HashiCorp Vault with the following sample key-value structure and update with actual values:

{
  "CONNECT_BOOTSTRAP_SERVERS": "",
  "CONNECT_CONSUMER_SASL_JAAS_CONFIG": "",
  "CONNECT_CONSUMER_SASL_MECHANISM": "",
  "CONNECT_CONSUMER_SECURITY_PROTOCOL": "",
  "CONNECT_PRODUCER_SASL_JAAS_CONFIG": "",
  "CONNECT_PRODUCER_SASL_MECHANISM": "",
  "CONNECT_PRODUCER_SECURITY_PROTOCOL": "",
  "CONNECT_SASL_JAAS_CONFIG": "",
  "CONNECT_SASL_MECHANISM": "",
  "CONNECT_SECURITY_PROTOCOL": ""
}

Procedure

  1. Configure chart-location/cdp-s3-sink-connector/values.yaml with the environment-specific configuration.
  2. Enable ConfigMaps and configure the required CDP S3 Sink Connector application URLs.
  3. Update the CDP S3 Sink Connector container image repository and required image version/tag.
  4. Verify the image pull secret exists.
    kubectl get secret hclcr-cred -n cdp-dev-app
  5. Enable Ingress and configure the required hostname, path, and TLS settings.
  6. Ensure the vault-creds Secret exists.
    kubectl get secret vault-creds -n cdp-dev-app
  7. The init container retrieves the s3-sink-connector secret from CDP_Secrets and writes it to /etc/secrets/mysecrets.json.
  8. Verify or create the namespace.
    kubectl get namespace cdp-dev-app

    If the namespace does not exist, create it:

    kubectl create namespace cdp-dev-app
  9. Install CDP S3 Sink Connector using Helm.
    helm install cdp-s3-sink-connector \
      chart-location/cdp-s3-sink-connector \
      -f chart-location/cdp-s3-sink-connector/values.yaml \
      -n cdp-dev-app
  10. Verify the Helm release status.
    helm list -n cdp-dev-app

    Expected status: deployed.

  11. Verify the deployment and pods.
    kubectl get deployment -n cdp-dev-app
    kubectl get pods -n cdp-dev-app -l k8s-app=cdp-s3-sink-connector

    Expected pod status: READY 1/1, STATUS Running.