Journey Engine Cluster Configuration

In production systems, Kafka is typically run as a cluster. This allows better load management and handling capabilities as well as providing fault tolerance.

The Journey application, and the Journey Engine specifically utilizes this ability of Kafka to be setup in a cluster for high volume production systems which also need redundancy in case of the failure of a single node. In this document we detail out the proposed layout and required configuration to host Journey Engine on a typical production cluster. The figures and number are indicative and will need to change as per actual requirements.

This is applicable only for Linux systems since installing and running Kafka on Windows is not recommended and has known issues.
  • There is single shared Database Server for all applications and nodes.
  • Servers 01 – 03 run combinations of Journey Engine and Kafka.
  • It is recommended that the setup has a minimum of 2 instances of Journey Engine and 3 Kafka brokers.
  • Server 04 runs Platform, Journey Web and Zookeeper. If other applications are also required and installed on this Server, the configuration shall need to be adjusted accordingly.
Typical Server Configuration
  1. Journey Engine
    • CPU – 8 cores
    • RAM – 32 GB
    • Disk Space – 250 GB
  2. Kafka
    • CPU – At least 4 cores
    • RAM – 32 GB
    • Disk Space – 500 GB
  3. Journey Web + Platform
    • CPU – 4 cores
    • RAM – 32 GB
    • Disk Space – 250 GB
  4. Database
    • CPU – 4 cores
    • RAM – 32 GB
    • Disk Space – 500 GB

Required Configurations for Cluster

Journey Engine
  1. Application properties
    • spring.ignite.ipFinder.List=< IPs of all system to be part of cluster>
      • Example - spring.ignite.ipFinder.List =10.115.36.20,10.115.36.21,10.115.36.22
      • This is to be done on all nodes where Journey Engine is running
      • If new node is to be added in cluster then IP should be added to this property on all servers and nodes to be restarted.
    • journey.ignite.reload=<set to all, will reload all active journeys>
      • Example - journey.ignite.reload=all
      • Example - journey.ignite.reload=all This property is to be enabled on Master Node – the node that is started first and shutdown last - only
      • Other Node(s) will get replicated data
    • journey.ignite.cluster=true – on all nodes
    • Journey.engine.secondary.node -- this property will have value as false for primary node, for rest of Engine nodes this property will have value as true.
    • Journey.cache.sync property to be enabled
      • Only on Master Node or any one node which will be started first and closed in last
Kafka
  1. Before starting Zookeeper and Kafka – increase ( ulimit -n 100000, on linux )
  2. Zookeeper port (2181 default) should be accessible from all servers in cluster
    • Test by using telnet <IP> 2181
  3. Broker port (default 9092) should be accessible from all servers where engine is running
  4. After starting Zookeeper, use the following command to check status if application is up and port is occupied
    • netstat –tulnp | grep 2181
  5. Each Broker should point to same zookeeper
  6. Under Broker property set replication to minimum 2,
    • If cluster contains 3 servers set the following values
      1. offsets.topic.replication.factor=2
      2. transaction.state.log.replication.factor=2
      3. transaction.state.log.min.isr=2
    Note: When user wants to use multiple kafka brokers and they start their engine with single kafka borker, then topics by default get created on this single broker. Later, when user updates the broker list then kafka doesn’t replicate the topics on other brokers because at the time of installation the replication value was set to one. Hence, during installation only, user needs to specify whether they want to use kafka with multiple brokers or with single broker.

    Cluster Kafka:

    Kafka Broker list: <BROKER_HOST1>:<PORT>, <BROKER_HOST2>:<PORT>, <BROKER_HOST3>:<PORT>

    Information message on the broker list format.

    What it shall do:

    1. Update engine and web application properties with below parameter

      spring.kafka.bootstrap-servers=<BROKER_HOST1>:<PORT>, <BROKER_HOST2>:<PORT>, <BROKER_HOST3>:<PORT>

    2. JourneyEngine :

      We added below property in journey_master_config.properties file in Journey Engine. Need to update from installer according to number of broker.

      journey.kafka.replications = <Number of Brokers>

    3. JourneyWeb :

      We added below property in JounreyWeb/application.properties file in Journey web. Need to update from installer according to number of broker.

      journey.kafka.customtopic.replications = <Number of Brokers>

Starting Journey Engine

The Journey Engine depends on Kafka and Ignite. The lifetime of Ignite is managed by Engine but Kafka needs to be started and stopped along with Journey Engine in a specific sequence to avoid Data corruption
  • Start Zookeeper
  • Start Kafka brokers on all servers
  • Start Journey Engine nodes in order
  • The Engine node that is started first – the Master Node – should be the last node whilst shutting down. This is important for Ignite Cluster to avoid corruption of cached data