Event Routing

Overview of Event Routing

Event Routing controls how incoming customer events move through the HCL CDP. The routing pipeline evaluates each event, applies transformation rules if required, and delivers the event to the correct destination such as Kafka, Amazon S3, or HCL CDP Unified Profiles (SST).

Routing decisions are based on the event's type and names, and they determine which path the event takes:

  • SST Path: Sent to the HCL CDP Unified Profiles.
  • S3: Sent directly to long-term storage (Amazon S3) for analytics, bypassing the SST path.
  • Real-Time Engagement: Routed to systems like Unica Journey via a Kafka topic.
  • Drop Event: Dropped altogether if the event is not relevant.

Event Routing Workflow

Event routing is orchestrated by an Apache NiFi pipeline that functions as the central nervous system for data flow. The process follows four key steps:

  1. Create a Source: Define the entry point for the events, such as the DI API or the Pixel Listener.
  2. Create a Destination: Define the output system, such as Kafka, Amazon S3, or the HCL CDP Unified Profile (SST).
  3. Link Source to Destination: Assign the available destinations to the newly created source to establish the data flow path.
  4. Apply Event Dictionary Rules: Define granular routing control based on specific Event Name or Event Type.

Custom Events (Type: TRACK)

For Custom Events, routing is managed through the user interface:

  1. Create the custom event in the Event Dictionary.
  2. Assign Allowed Destinations for that specific event name.

Standard Events (Type: PAGE, IDENTIFY)

Allowed destinations for Standard Events are configured through backend mappings in the VRM database.

Use Event Transformation

Event Transformation allows you to restructure or modify the event payload before it is routed to a destination.

Transformation rules convert the raw incoming payload into the specific format expected by the destination system. This is required because different destinations often need different data formats.

The CDP uses the FreeMarker Template Language (FTL) to apply custom transformation logic within the NiFi pipeline. FTL templates enable you to:

  • Rename fields (e.g., properties.FirstName to first_name).
  • Restructure the payload (e.g., convert a nested JSON into a flattened structure).
  • Filter unnecessary attributes (e.g., remove userAgent before sending to S3).
  • Apply custom business logic (e.g., add eventCategory: "Offer" conditionally).

Applying Transformation Rules

You can apply a transformation for a specific event and destination in the Event Dictionary:

  1. In the Event Transformation tab, click Add Transformation.
  2. Select the Event and the Destination.
  3. Assign the FTL Template that will be applied for that specific event-destination pair. For more information about Event Transformation, refer Managing Event Transformation.

k