Deploying HCL CDP Audience Uploader

This section provides detailed instructions on how to deploy HCL CDP Audienceuploader using the Devtron in the AWS.

Before you begin

Make sure to create cdp-Audienceuploader secret in the AWS secret manager before deploying CDP Audience Uploader.

To create CDP Audienceuploader secret in the AWS secret manager , follow the steps below:

  1. Create a cdp-Audienceuploader secret sample key and value, and update the actual values as shown below.
    {
  "GANGLIA_COLLECTORS": "",
  "GANGLIA_PORT": "8650",

  "CONSUMER_TOPIC_LIST": "dmp_rt_segments",
  "KAFKA_TOPIC_LIST": "dmp_rt_segments",
  "KAFKA_BATCH_SIZE": "100",
  "KAFKA_TOPIC_CONSUMERS": "6",
  "KAFKA_DEFAULT_TIMEOUT": "1000",
  "KAFKA_DEFAULT_TRIGGER": "5000",
  "CONSUMER_GROUP": "",
  "ZOOKEEPER_LIST": "",

  "KAFKA_CONSUMER_CONFIG": "bootstrap.servers::<KAFKA_HOST_1>:9096,<KAFKA_HOST_2>:9096|security.protocol::SASL_SSL|sasl.jaas.config::org.apache.kafka.common.security.scram.ScramLoginModule required username='<KAFKA_USER>' password='<KAFKA_PASSWORD>';|sasl.mechanism::SCRAM-SHA-512|client.dns.lookup::use_all_dns_ips|group.id::k8s-audiencemanager-prod",

  "DB_HOST": "<DB_HOST>",
  "DB_PORT": "3307",
  "DB": "vrm",
  "DB_USERNAME": "<DB_USER>",
  "DM_PASSWORD": "<DB_PASSWORD>",

  "SQS_QUEUE_URL": "https://sqs.<AWS_REGION>.amazonaws.com/<ACCOUNT_ID>/<QUEUE_NAME>",

  "AEROSPIKE_HOST": "<AERO_HOST_1>,<AERO_HOST_2>",
  "AEROSPIKE_PORT": "13000",
  "APS1_HOST": "<AERO_HOST_1>,<AERO_HOST_2>",

  "KafkaProducerConfig": "bootstrap.servers::<KAFKA_HOST_1>:9096,<KAFKA_HOST_2>:9096|security.protocol::SASL_SSL|sasl.jaas.config::org.apache.kafka.common.security.scram.ScramLoginModule required username='<KAFKA_USER>' password='<KAFKA_PASSWORD>';|sasl.mechanism::SCRAM-SHA-512|client.dns.lookup::use_all_dns_ips|acks::all|max.block.ms::5000",

  "KafkaProducerPoolSize": "10",
  "KafkaProducerMeta": "true",
  "KafkaProducerHostname": "cdp-audience-manager",
  "KafkaRegion": "aps1",

  "6525_ApiCtx": "<API_KEY>",
  "6526_ApiCtx": "<API_KEY>",
  "6534_ApiCtx": "<API_KEY>",
  "6530_ApiCtx": "<API_KEY>",
  "6529_ApiCtx": "<API_KEY>",
  "6528_ApiCtx": "<API_KEY>",
  "6527_ApiCtx": "<API_KEY>",

  "CLIENT_ID": "<CLIENT_ID>",
  "CLIENT_SECRET": "<CLIENT_SECRET>"
}
  2. Update ConfigMaps data with actual values, and create required Kafka topics before deployment.

Procedure

  1. Navigate to the Devtron Chart Store, and select the cdp-Audienceuploader chart to deploy.
  2. Now, configure and deploy the cdp-Audienceuploader charts.
  3. In the YAML section, update the ConfigMap using the below details and deploy the chart. 
    OPENTSDB_HOST: hxcb-cdp-opentsdb-service.cdp-dev-app.svc.cluster.local
OPENTSDB_PORT: "80"
GANGLIA_PUSH_METRIC_FREQUENCY: "120"
PR_DELAY: "3600"
PR_PERIOD: "3600"
ZOOKEEPER_TIMEOUT: "60000"
ZOOKEEPER_SYNC_TIME: "2000"
AUTOCOMMIT_INTERVAL: "1000"
ZOOKEEPER_CONNECTION_TIMEOUT: "60000"
CONSUMER_TIMEOUT: "1000000"
SOCKET_RECEIVE_BUFFER: "2097152"
FETCH_MESSAGE_MAX: "10485760"
REBALANCE_BACKOFF_MS: "10000"
SOURCE_DIRECTORY_PATH: /disk1/CDPAudienceUploader/extractor/processed
DESTINATION_DIRECTORY_PATH: /disk1/CDPAudienceUploader/extractor/uploaded
INVALID_DIRECTORY_PATH: /disk1/CDPAudienceUploader/extractor/invalid
EXTRACTOR_BOOTSTRAP_WAIT: "2"
NEW_FILE_WAIT: "1800"
MAX_POOL_SIZE: "20"
CORE_POOL_SIZE: "10"
EMAIL_PARAMS: |
  {}
MOBILE_PARAMS: |
  {}
DEVICEID_PARAMS: |
  {}
EXTERNALID: |
  {}
ACTIVE_CAMPAIGNS: "6525"
DB_DRIVER: org.mariadb.jdbc.Driver
CRON_EXPR: 0 */5 * * * ?
AEROSPIKE_NAMESPACE: cdpstore
AEROSPIKE_EXPIRY: "31536000"
AEROSPIKE_TIMEOUT: "3000"
SUPPORTED_REGIONS: aps1
APS1_PORT: "13000"
APS1_NAMESPACE: cdpstore
KMS_REGION: ap-south-1
REGION: ap-south-1
batchsize: "1"
KMS_IMPLEMENTATION: AWS
  4. On successful deployment, validate the deployment as shown below.