Deploying CDP S3 Sink Connectors

This section provides detailed instructions on how to deploy HCL CDP S3 Sink Connectors via the Kafka Connect REST API.

Procedure

  1. Identify the S3 Sink Connector pod in the target namespace. 
    kubectl get pods -n <namespace> | grep s3-sink-connector
  2. Open a shell session in the S3 Sink Connector pod. 
    kubectl exec -it cdp-s3-sink-connector-bf4bd5ddf-qvlnx -- sh -n <namespace>
  3. Run curl requests to create the S3 Sink connectors. Replace aws.secret.access.key, aws.access.key.id, and store.url with environment-specific values.

    Create the s3-profile-data-sync-6550 connector:

    curl --location 'http://localhost:8083/connectors'   --header 'Content-Type: application/json'   --data-raw '{
    "name": "s3-profile-data-sync-6550",
    "config": {
      "connector.class": "io.confluent.connect.s3.S3SinkConnector",
      "s3.region": "ap-south-1",
      "tasks.max": "2",
      "topics": "mongo-data-sync-6550",
      "store.url": "https://cdp-minio-api.<domain-name>",
      "aws.secret.access.key": "",
      "aws.access.key.id": "",
      "s3.bucket.name": "hclsw-s3-cdp-app",
      "s3.path.style.access": "true",
      "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
      "storage.class": "io.confluent.connect.s3.storage.S3Storage"
    }
  }'
  4. Verify the connector status. 
    curl http://localhost:8083/connectors/s3-sink-connect-diapi-Events/status
curl http://localhost:8083/connectors/s3-profile-delete-sync/status
curl http://localhost:8083/connectors/s3-profile-data-sync-6550/status