Feed Applications Models

This page explains how feed application models in HCL Detect define the structure, enrichment logic, and aggregation behavior for real-time data feeds. It covers configuration files, enrichment functions, aggregations, and integration with data models and profiles.

Each feed application have to configure application model and can optionally configure some other feed application model file as described below.

feed application models are configured in etc/models/applications/<feed-name>/.

below is the example of a solution containing a feed application definition:

$ tree etc/model/applications/ericsson_usage/
etc/model/applications/ericsson_usage/
��������� aggregations.json
��������� application.json
��������� enrichment_functions.json
��������� enrichments.json

The name of feed application is Ericsson Usage .

Below sub-sections will explain each of the above files.

Application Model

Application model defines the key attribute and feed data model that this feed implements, and it is defined in application.json file.

Example application Model:

{
    "feedApplication": {
        "dataModels": [
            "Identity",
            "Usage"
        ],
        "keyAttribute": "MSISDN",
        "name": "Ericsson Usage"
    }
}

dataModels a list of feed data model that are implemented by this feed.
keyAttribute an attribute from the feed that is a key attribute.
name is name of feed application.

Aggregations

This model describes the aggregations that are computed on a feed and stored in FastPast. Aggregates are configured in etc/model/applications/<application_name>/aggregation.json file.

Example Aggregation definition is as below:

{
    "aggregations": [
        {
            "name": "avgCallDuration",
            "operation": {
                "aggregationKind": "Average",
                "groupByAttributes": [
                    "callingNumber"
                ],
                "valueAttribute": "duration"
            }
        },
        {
            "name": "numCallsByCell",
            "operation": {
                "aggregationKind": "Sum",
                "groupByAttributes": [
                    "cellId"
                ],
                "value": 1.0
            }
        },
        ..
        ..
    ]
}

aggregations List of aggregations.
name A name (aggregations defined on different feeds can have the same name) that is in the form of an identifier (aNameLikeThis).
groupByAttributes A potentially empty list of group by attributes.
aggregationKind An aggregation kind (Average, Maximum, Minimum, Sum, Variance).
valueAttribute or value A value (such as "1" for counting) or value attribute (such as "callDuration").
tupleFilter An optional tupleFilter, which is a Boolean UEL expression

Loaded once during Detect initialization, managed from the UI there on.

Enirchment Functions

Enrichment Function model file defines the meta data of an enrichment function. It defines the signature of a python function to be used in an enricher.

Enrichment Function definition is configured in etc/model/applications/<application_name>/enrichment_functions.json file.

Example Enrichment Function:

{
    "functions": [
        {
            "module": "acme.application_helpers.ericsson_usage.enrichment_helpers",
            "name": "device_change",
            "parameters": [
                {
                    "name": "deviceName",
                    "required": false,
                    "type": "String"
                },
                {
                    "name": "isSmartphone",
                    "required": false,
                    "type": "Bool"
                }
            ],
            "type": "Bool",
            "usedAttributes": [
                {
                    "name": "IMEI",
                    "required": true,
                    "type": "String"
                },
                {
                    "name": "lastIMEI",
                    "required": true,
                    "type": "String"
                }
            ]
        },
        ..
        ..
    ]
}

signature of python function is as below:

def device_change_(tuple_, deviceName=None, isSmartphone=None):
    """Returns the last time the subscriber has changed their device"""
    imei = tuple_.IMEI
    lastIMEI = tuple_.lastIMEI
    ..
    ..
    return xyz

functions list of functions defined.
module a python module path.
name name of enrichment function.
type return type of the function.
parameters list of additional parameters to enrichment function, tuple is a default and first parameter to function. we need to only mention any additional parameters/
usedAttributes the list of attributes used from tuple. This is required for Detect to know if a particular function can be applied on a feed or not.

Enrichments

This model describes the enrichments that are performed on a feed, whose results may be stored in PinPoint. Enrichments are configured in etc/model/applications/<application_name>/enrichments.json. It has list of enrichments, where each enrichment has: * A name (enrichments defined on different feeds can have the same name)

A list of transformed attributes
A list of lookup attributes
A list of aggregated attributes
A list of derived attributes

There are two kinds of enrichments: Pre-aggregation are list of enrichments that are applied before aggregation takes place. Post-aggregation are list of enrichments that are applied after aggregation takes place.

This file is loaded once during Drive initialization, managed from the UI there on.

Enrichment structure is as below:

{
    "preAggregationEnrichments": [
        {
            "name": "IMEIChangeEnrichmentPre",
            "operation": {
                "derivedAttributes": [
                    ..
                    ..
                ],
                "aggregatedAttribute": [
                    ..
                    ..
                ],
                "lookupAttribute": [
                    ..
                    ..
                ],
                "transformedAttributes": [
                    ..
                    ..
                ]
            }
        },
        {
            ..
        }
    ],
    "postAggregationEnrichments": [
        {
            "name": "IMEIChangeEnrichmentPost",
            "operation": {
                "derivedAttributes": [
                    ..
                    ..
                ],
                "aggregatedAttribute": [
                    ..
                    ..
                ],
                "lookupAttribute": [
                    ..
                    ..
                ],
                "transformedAttributes": [
                    ..
                    ..
                ]
            }
        },
        {
            ..
        }
    ]
}

Transformed Attributes

Its an UEL expression to derive a value for a new attribute.

Example:

 {
    "expression": "\"FOO\"",
    "name": "identity",
    "retained": false,
    "type": "String"
}

expression UEL expression for to derive value.
name name of enriched attribute.
retained flag to retain this new attribute in the tuple for next operator in the flow.
type is the data type of this attribute.

Lookup Attributes

This is used to look-up from a profile.

Example:

{
    "name": "licenseId",
    "retained": false,
    "tableAttribute": {
        "keyAttribute": "identity",
        "name": "licenseId",
        "table": "CUSTOMER"
    },
    "type": "String"
}

tableAttribute is to define the table and attribute to look-up.
tableAttribute.keyAttribute is key attribute in the tuple which will be used to perform lookup from table.
tableAttribute.name is the name attribute from table to be looked up.
tableAttribute.table is the table name form pinpoint to be looked up.
type is the data type of the enrichmented attribute.

Aggregated Attributes

This enrichment attribute gets value by fetching a aggregate from FastPast.

Example:

{
    "aggregate": "sumOfTransactionAmountByCustomerAndTransactionType",
    "groupByAttributes": [
        "customerId",
        "transactionType"
    ],
    "name": "transactionAmountInLast7Days",
    "period": "Last",
    "retained": true,
    "type": "Double",
    "windowLength": 7,
    "windowLengthUnit": "Day"
}

aggregate is the name of aggregate in the FastPast.
groupByAttributes is a list of attributes to be used for passing for groupby attributes.
name is the name of enriched attribute.
period is the period for FastPast query. e.g. Current, Last or AllTime.
windowLength is length of window for a given unit.
windowLengthUnit is the unit for which query to be made. e.g., Day, Month, Year, Minute, Hour

Derived Attributes

The derived attributes enrichment enables the execution of an external Python function. One such a function takes in a tuple (as well as any other required additional parameters, if any), performs a user-defined computation, and produces a result. This function invocation's return value can then be retained and forwarded as part of an outgoing tuple.

Example:

{
    "function": {
        "module": "acme.application_helpers.ericsson_usage.enrichment_helpers",
        "name": "device_change"
    },
    "name": "deviceChanged",
    "retained": true,
    "type": "Bool"
},
{
    "function": {
        "module": "acme.application_helpers.ericsson_usage.enrichment_helpers",
        "name": "device_change_time"
    },
    "name": "deviceChangeTime",
    "retained": true,
    "storeBackAttribute": {
        "keyAttribute": "callingNumber",
        "name": "lastIMEIModificationTime",
        "table": "CUSTOMER_PROFILE"
    },
    "type": "Int64"
},

function This refers to enrichment function model discussed in previous sub section.
storeBackAttribute the derived attribute can optionally stored back to profile in pinpoint.