Kafka

This page provides step-by-step instructions to configure Kafka as a destination in HCL CDP, enabling the consumption of raw events directly to your Kafka consumer.

Kafka integration allows HCL CDP to stream raw events from various data sources to your configured Kafka destination. This enables seamless real-time event processing and data streaming for further analysis and applications.

Setup Kafka as Destination in HCL CDP

  1. Navigate to Data Pipeline > Destinations, and click +Create Destination.

  2. Select Kafka from the destination list.
  3. Choose the source you created earlier, and click Continue.
  4. In the Destination Name field, enter a unique name for your destination, and click Submit.
  5. After creating the destination, navigate to the CONFIGURATION tab.

    Configuring Kafka as destination.

  6. Enter the following details to configure your destination.
    • Name: Enter a descriptive name for your Kafka destination.
    • Hostname: Provide the Kafka host address where your Kafka instance is hosted.

    • Authentication: choose between PLAINTEXT and SASL_PLAINTEX. If using SASL_PLAINTEXT, select one of the three available authentication mechanisms.

    • Topic name: Enter the Kafka topic name where HCL CDP will produce the Kafka events.
  7. Click Test to verify the connection and ensure events can be produced to the specified Kafka configuration. If successful, a success message will appear. If not, an error message will indicate the issue.
  8. After a successful test, click Update to save your Kafka destination configuration.
  9. Toggle the activation status to enable the destination.
    Tip: It may take up to 30 minutes for the destination to become fully functional after activation.