Audience Ingestion Hints

The following section explains the use of audience ingestion hints for optimizing audience entries into Journeys configured with audience deduplication setting. If the Journey is configured for audience deduplication, Journey engine needs to look up every incoming audience record to decide if it is a new audience or an existing one. As per the lookup result, it either inserts a new audience record or performs deduplication on existing record. This approach has an overhead of consulting the audience database for every single incoming message. Consequently, this slows down audience ingestion process, which leads to delayed Journey processing. With the intention to optimize audience deduplication process, Journey engine supports certain hints, which can be supplied along with the audience message.
Note: Following hints are ignored for the Journeys that have not been configured with deduplication setting. Furthermore, these hints are supported only for Kafka & REST entry sources.

Generally, the audience message for REST & Kafka entry sources would look like below JSON:

{
    "entrySourceCode": "ES-00000001",
    "entrySourceType": "KAFKA",
    "data": [
        {
            "email": "email1@domain.com",
            "mobile": "1122334455",
            "birthdate": "09 01 1985",
        },
        {
            "email": "email2@domain.com",
            "mobile": "5544332211",
            "birthdate": "01 09 1995",
        },
        { 
            "email": "email3@domain.com", 
            "mobile": "5566778899", 
            "birthdate": "05 11 2000", 
        }
    ]
}

This JSON can now carry an additional hint to convey existence of given audience(s). Based on the hint, Journey engine can skip the audience existence check, and directly go for new audience insertion or deduplication updates. Accordingly, above message can carry single hint common to all the audiences(3) included in the message.

For example,

{
    "entrySourceCode": "ES-00000001",
    "entrySourceType": "KAFKA",
    "__hints":{ 
        "action": "U" // I -> Insert, U -> Update 
    },
    "data": [
        {
            "email": "email1@domain.com",
            "mobile": "1122334455",
            "birthdate": "09 01 1985",
        },
        {
            "email": "email2@domain.com",
            "mobile": "5544332211",
            "birthdate": "01 09 1995",
        },
        { 
            "email": "email3@domain.com", 
            "mobile": "5566778899", 
            "birthdate": "05 11 2000", 
        }
    ]
}
The action property inside the __hints object conveys the expected action for the audiences. The action property can take one the following values, and their impact is as mentioned below:
action property value Impact on audience ingestion
I

Audience(s) will be considered completely new and will be blindly inserted into the audience database.

Caveat ��� System will not check if the audiences are truly new or not. Hence, any erroneous hint may lead to the redundant record(s) of same audience if it is sent with ���I��� hint more than once.
U

Audience(s) will be deemed pre-existing, and system will update the existing records with newly received changes.

Caveat ��� System will not check if there is really any existing audiences based on the deduplication fields. If no such audiences are present, then the updates will have no impact. System will not convert updates into new audience insertions.
null or missing If the property value is set to null and/or if the hint itself is missing, then Journey engine will consult audience database to see if incoming audiences already exist or not. Accordingly, it will perform either audience insertion or modification. In a nutshell, it will continue to run its normal course.

Like the common audience hint, above JSON message can alternatively carry separate hint for each audience. For example: 

{
    "entrySourceCode": "ES-00000001",
    "entrySourceType": "KAFKA",
    "data": [
        {
            "__hints":{ 
                "action": "U" // I -> Insert, U -> Update 
            },
            "email": "email1@domain.com",
            "mobile": "1122334455",
            "birthdate": "09 01 1985",
        },
        {
            "__hints":{ 
                "action": "I" // I -> Insert, U -> Update 
            },
            "email": "email2@domain.com",
            "mobile": "5544332211",
            "birthdate": "01 09 1995",
        },
        { 
            "email": "email3@domain.com", 
            "mobile": "5566778899", 
            "birthdate": "05 11 2000", 
        }
    ]
}

The impact of this approach is exactly similar to the common hint approach explained earlier. For the above example, first audience will be modified with the new details, second will be inserted as a new audience, whereas for the third audience, Journey engine will first check its existence & work accordingly.

Additionally, combination of common & separate hints is also supported. For example, 

{
    "entrySourceCode": "ES-00000001",
    "entrySourceType": "KAFKA",
    "__hints":{ 
        "action": "I" // I -> Insert, U -> Update 
    },
    "data": [
        {
            "__hints":{ 
                "action": "U" // I -> Insert, U -> Update 
            },
            "email": "email1@domain.com",
            "mobile": "1122334455",
            "birthdate": "09 01 1985",
        },
        {
            "email": "email2@domain.com",
            "mobile": "5544332211",
            "birthdate": "01 09 1995",
        },
        { 
            "email": "email3@domain.com", 
            "mobile": "5566778899", 
            "birthdate": "05 11 2000", 
        }
    ]
}

For the above example, common hint (I) will be applicable to the second & third audience records. Thereby, second & third audiences will be treated as new audiences & shall be inserted into the database. First audience however carries an overridden hint, accordingly to which Journey engine will perform deduplication update for it.