Configuring Apache Kafka topics manually

Before you begin

To create the topics manually, you must have access to an administration machine with the Apache Kafka command-line utilities installed and network connectivity to your cluster brokers. You must also have the full list of required topics available from the Kafka topic reference.

About this task

In highly secure or enterprise production deployments, automatic topic creation is often disabled (auto.create.topics.enable=false) to protect cluster performance and maintain naming conventions. In these environments, a Kafka administrator must provision all required topics before deploying the HCL Universal Orchestrator services.

If the optional kafka.prefix parameter is set in the values.yaml file of the hcl-uno-chart, its value must be prepended to every topic name when creating topics manually. A topic prefix enables multiple HCL Universal Orchestrator instances to share a single Apache Kafka server without topic name collisions.

If HCL Universal Orchestrator is configured without a Kafka admin user with a default configuration, set kafka.configureTopicOnStart to false in the values.yaml file. This prevents HCL Universal Orchestrator from attempting to create or update topics on startup.

Procedure

Configure the Kafka connection settings in the values.yaml file of the hcl-uno-chart.

The following parameters control the connection between HCL Universal Orchestrator and your Apache Kafka cluster:

kafka.url

The URL of the event streaming platform. Mandatory parameter.

kafka.prefix

An optional prefix prepended to every topic name. The default value is empty. When set, use the same value as the prefix when creating topics manually.

kafka.username

The name of the Kafka user. Mandatory parameter.

kafka.password

The password of the Kafka user. Mandatory parameter.

kafka.tls

Specifies whether TLS is enabled. Supported values are true and false.

kafka.tlsInsecure

Specifies whether certificate validation is skipped. Supported values are true and false.

kafka.saslMechanism

The SASL mechanism for the Kafka client configuration.

kafka.jaasConfig

The JAAS configuration for the Kafka client.

kafka.securityProtocol

The security protocol for the Kafka client configuration.

kafka.kerberosServiceName

The Kerberos service name.

kafka.oauthLoginCallbackHandlerClass

The OAuth login callback handler class.

kafka.oauthTokenEndpointUrl

The OAuth token endpoint URL.

kafka.topicReplicas

The number of replicas for each topic. The default value is 1. Set this value to match the --replication-factor parameter used when creating topics manually.

kafka.configureTopicOnStart

Specifies whether HCL Universal Orchestrator creates or updates topics on startup. Supported values are true and false. Set to false when Kafka is configured without an admin user or when topics are managed manually.
Log in to your administration machine and run the kafka-topics.sh creation command for each topic listed in the Kafka topic reference.
Use the following template for each topic:
```
kafka-topics.sh --bootstrap-server kafka-broker:port \
  --create \
  --partitions partitions \
  --replication-factor 3 \
  --topic prefixtopic-name
```
Replace kafka-broker and port with the address of your bootstrap server, topic-name with the base topic name from the reference table, and partitions with a value equal to 3 times the maximum number of product pod instances in your deployment. Replace prefix with the value of kafka.prefix from your values.yaml file, or omit it if kafka.prefix is not set.
Apply sizing and tuning recommendations based on the expected traffic for each topic.
While a replication factor of 3 is standard for high-availability enterprise environments, traffic volume varies per topic. Apply the following optimization rules during configuration:
State-tracking topics
For topics such as consumed-jobs and task-status-update, you can optionally apply a log compaction policy and a minimum retention window to manage disk utilization over long runtimes:
```
--config cleanup.policy=delete --config retention.ms=172800000
```
The retention.ms value of 172800000 corresponds to 48 hours. Increase this value if your environment requires a longer retention window.
Query your bootstrap server to verify that all required topics are visible and healthy before proceeding with the application installation.
```
kafka-topics.sh --bootstrap-server kafka-broker:port --list
```
Confirm that the output includes every topic listed in the Kafka topic reference.
All required topics are present in the cluster and ready for use.

Results

All required Apache Kafka topics are created and available in the cluster. You can proceed with the deployment of HCL Universal Orchestrator.

What to do next

After the topics are created, continue with deploying the HCL Universal Orchestrator services. For deployment steps, see Deploying HCL Universal Orchestrator.