Downstream Kafka based Custom Event Consumer Configuration

Kafka is a distributed event store and stream-processing platform that captures events and send them to custom event application. Using the Kafka-based configurations, customers can implement this configuration and use the Kafka route to handle their detected events.

For publishing events to downstream Kafka which is read by downstream system, follow the steps below:
  1. In the acme/etc/producer_config.properties file, update the IP address of Kafka server.
  2. In the drive.json file, update the corresponding parameters in the eventConsumers section:
    1. the className parameter with the name of the Java class.
    2. the parameters > name parameter with name of the downstream system.
    3. the producerTopic value with the downstream system Kafka topic name.
  3. Update the following database parameters values with the values from the drive.json file:
    1. databaseServerIp: update it with database server IP address.
    2. databaseServerPort: update it with database server port.
    3. databaseServerUser: update it with database server userName.
    4. databaseServerPassword: update it with database server userPassword.
  4. Add other downstream specific parameter to this sample json.
  5. In the eventEndpoints parameters, add the endpoint name.
    Note: The value of APPLICATION_ENDPOINT_NAME should match with the event consumer class.
  6. After updating the values, ensure maven is installed on the server.
To build the jar for the custom_event_consumer, follow the steps below:
  1. Navigate to the HCL Detect installation folder.
  2. Change directory to custom_event_consumer using the below command.
    cd custom_event_consumer/
  3. By default, the custom_event_consumer folder contains the following makefile, which to can create and move the custom event consumer jar file to installer path "/tomcat/webapps/drive/WEB-INF/lib/".
    .PHONY: build move clean all
all: build move clean
build: 
	mvn -f pom-custom.xml clean install
move:
	cp target/custom_event_consumer-*.jar $(HCL_INSTANCE_HOME)/tomcat/webapps/drive/WEB-INF/lib/
clean:
	rm -rf target
  4. To execute the makefile, use the following command.
    “make”
  5. On Successful executing the makefile, the jar of the custom_event_consumer is moved to tomcat.
The sample configuration of downstream Kafka publishing events is shown below:
"eventConsumers": [
	{
		"className": "com.detect.SampleKafkaBasedEventConsumer",
		"name": "SampleKafkaBasedEventConsumer",
		"parameters": [
			{
				"name": "payloadEntrySourceCode",
				"stringValue": "ES-TEST002"
			},
			{
				"name": "databaseServerIp",
				"stringValue": "127.0.0.1"
			},
			{
				"name": "databaseServerUser",
				"stringValue": "drive"
			},
			{
				"name": "databaseServerPassword",
				"stringValue": "6F2FA7F6B3CD1570F19F0A0B2636E033"
			},
			{
				"name": "databaseServerPort",
				"int32Value": 3306
			},
			{
				"name": "databaseName",
				"stringValue": "drive_acme_core"
			},
			{
				"name": "producerConfigFilePath",
				"stringValue": "${HCL_HOME}/../${HCL_SOLUTION}/etc/producer_config.properties"
			},
			{
				"name": "producerKrb5ConfigFilePath",
				"stringValue": "/etc/krb5.conf"
			},
			{
				"name": "producerTopic",
				"stringValue": "CUSTOM_QUALIFIED_TRANS_TOPIC"
			}
		]       
	}
]
  1. eventConsumers.className: set the name of the Java class responsible for writing events in the pre-defined format of the downstream system.
  2. eventConsumers.name: name of the downstream system intended to assign.
  3. Parameters:
    1. Database related:
      1. databaseServerIp: Server IP of the database to publish downstream events to.
      2. databaseServerUser: Username of the database credentials.
      3. databaseServerPassword: Encrypted password of the database credentials.
      4. databaseServerPort: Port of the database.
      5. databaseName: Name of the database.
    2. Kafka related:
      1. producerConfigFilePath: Path to the producer_config.properties file. This file contains the configuration/properties for connecting to Kafka.
      2. producerKrb5ConfigFilePath: Path to the krb5.conf file if needed to the Kafka connection needs to be Kerberos authenticated.
      3. producerTopic: Name of the Kafka Topic onto which events will be published and downstream system reads from.
Sample Message

This Java class writes the events in the pre-defined format of the downstream system to implement the EventConsumer interface and its methods.

  • initialize() - this method creates a database & downstream Kafka connection.
  • consumeEvent() - this method receives the event object.
  • consumeEventImpl() - this method sets database message, downstream Kafka message and writes the respective message to database and downstream Kafka topic.