Deploying CDP RTS

This section provides detailed instructions on how to deploy HCL CDP RTS using the Devtron in the AWS.

Prerequisites:

Make sure to create rts secret with required data in the AWS secret manager before deploying CDP RTS.

To create the rts secret in the AWS secret manager, follow the steps below:

  1. Create a rts secret sample key and value in the rts secret, and update ConfigMaps data with actual values.
    {
  "bootstrap.servers": "<KAFKA_HOST_1>:9096,<KAFKA_HOST_2>:9096",
  "security.protocol": "SASL_SSL",
  "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username='<KAFKA_USER>' password='<KAFKA_PASSWORD>';",
  "sasl.mechanism": "SCRAM-SHA-512",

  "MysqlHost": "<DB_HOST>",
  "MysqlDb": "vrm",
  "MysqlUser": "<DB_USER>",
  "MysqlPass": "<DB_PASSWORD>",
  "MysqlPort": "3307",

  "KafkaConsumerConfig": "bootstrap.servers::<KAFKA_HOST>|security.protocol::SASL_SSL|sasl.jaas.config::org.apache.kafka.common.security.scram.ScramLoginModule required username='<KAFKA_USER>' password='<KAFKA_PASSWORD>';|sasl.mechanism::SCRAM-SHA-512|client.dns.lookup::use_all_dns_ips|group.id::k8s-rts-prod",

  "KafkaProducerConfig": "bootstrap.servers::<KAFKA_HOST>|security.protocol::SASL_SSL|sasl.jaas.config::org.apache.kafka.common.security.scram.ScramLoginModule required username='<KAFKA_USER>' password='<KAFKA_PASSWORD>';|sasl.mechanism::SCRAM-SHA-512|client.dns.lookup::use_all_dns_ips|acks::1|linger.ms::100",

  "MONGO_URI": "mongodb://<MONGO_USER>:<MONGO_PASSWORD>@<MONGO_HOST_1>:51090,<MONGO_HOST_2>:51090,<MONGO_HOST_3>:51090,<MONGO_HOST_4>:51090/segmentation?authSource=admin"
}

Deploying CDP RTS

To deploy the CDP RTS, follow the steps below:

  1. Navigate to the Devtron Chart Store, and select the cdp-rts chart to deploy.

  2. Now, configure and deploy the CDP RTS charts.

  3. In the YAML section, update the ConfigMap using below details, and deploy the charts.
    APP_INFRA: k8s
APP_REGION: ap-south-1
APP_TYPE: realtime
EXECUTOR_SERVICE_MAX_THREADS: "32"
EXECUTOR_SERVICE_MIN_THREADS: "16"
EXECUTOR_SERVICE_QUEUE_SIZE: "1000"
EXECUTOR_SERVICE_THREAD_TTL: "5"
FROM_EMAIL: hcl.com

KAFKA_CONSUMER_GROUP_ID: k8s-rts-prod
KAFKA_POLL_INTERVAL: "200"
KAFKA_PRODUCER_FAILURE_TOPIC: dmp_rt_failure
KAFKA_PRODUCER_META: "true"
KAFKA_PRODUCER_POOL_SIZE: "16"
KAFKA_PRODUCER_RAMANUJAN_TOPIC: dmp_rt_ramanujan
KAFKA_PRODUCER_TOPIC: dmp_rt_segments
KAFKA_SST_TOPIC: dmp_sst_rt_profile
KAFKA_DETECT_AGGREGATE_TOPIC: CDP_AGGREGATED_TRANS_TOPIC
KAFKA_TOPIC_CONSUMERS: 6,6,6
KAFKA_TOPIC_CONSUMER_BATCH_SIZE: 100,100,100
KAFKA_TOPIC_CONSUMER_DEFAULT_BATCH_TIMEOUT_MS: 200,200,200
KAFKA_TOPIC_CONSUMER_DEFAULT_BATCH_TIMEOUT_TRIGGER: 500,500,500
KAFKA_TOPIC_LIST: dmp_sst_rt_profile,CDP_QUALIFIED_TRANS_TOPIC,CDP_AGGREGATED_TRANS_TOPIC

LOG_LEVEL_APP: error
LOG_LEVEL_METRICS: off
LOG_LEVEL_ROOT: error
LOG_LEVEL_UTILS: off

MAX_BATCH_EXECUTE_RETRIES: "3"
MAX_EVENT_BATCH_LAG: "64"

MODEL_CONFIG_LOCAL_PATH: /disk1/rts/config/predictiveSegmentationModel.ini
MODEL_CONFIG_S3_PATH: s3://<masked-bucket>/sst/predictivesegments/rtson/predictiveSegmentationModel.ini

MODEL_CUTOFFS_LOCAL_PATH: /disk1/rts/config/predictiveSegmentationCutoffs.ini
MODEL_CUTOFFS_S3_PATH: s3://<masked-bucket>/sst/predictivesegments/rtson/predictiveSegmentationCutoffs.ini

MYSQL_REFRESH_FAILURE_TOLERANCE_COUNT: "3"
MYSQL_REFRESH_INTERVAL: "120000"

PMML_MODEL_CONFIG_LOCAL_PATH: /disk1/rts/config/predictivesegments/pmml_model_config/
PMML_MODEL_CONFIG_S3_PATH: s3://<masked-bucket>/sst/predictivesegments/pmml_model_config/rtson/

PMML_MODEL_DIRECTORY_LOCAL_PATH: /disk1/rts/config/predictivesegments/pmml_models/
PMML_MODEL_DIRECTORY_S3_PATH: s3://<masked-bucket>/sst/predictivesegments/pmml_models/rtson/

TO_EMAIL: ""

TSDB_HOST: <internal-service>.svc.cluster.local
TSDB_PORT: "80"

AWS_REGION: ap-south-1
SNS_ALERT_ENDPOINT: ""

FROM_EMAIL_ID: user@domain.com
ALERT_TO_EMAIL_ID: user@domain.com

BUCKET_TYPE: s3
awsRegion: us-south-1

emailType: SES
fromEmail: user@domain.com

smtpHost: <smtp-relay-host>
smtpPort: "20225"
toEmail: user@domain.com

KAFKA_DETECT_TOPIC: CDP_QUALIFIED_TRANS_TOPIC
KMS_IMPLEMENTATION: AWS

LOOKUP_KEY_IDENTIFIER: '[{"VIZVRM6525":"crmid"},{"VIZVRM6526":"crmid"},{"VIZVRM6527":"crmid"},{"VIZVRM6528":"crmid"},{"VIZVRM6529":"crmid"},{"VIZVRM6530":"crmid"},{"VIZVRM6531":"crmid"},{"VIZVRM6532":"crmid"},{"VIZVRM6533":"crmid"},{"VIZVRM6534":"crmid"},{"VIZVRM6535":"crmid"},{"VIZVRM6536":"crmid"},{"VIZVRM6537":"crmid"},{"VIZVRM6538":"crmid"},{"VIZVRM6539":"crmid"},{"VIZVRM6540":"crmid"},{"VIZVRM6541":"crmid"},{"VIZVRM6542":"crmid"},{"VIZVRM6543":"crmid"},{"VIZVRM6544":"crmid"},{"VIZVRM6545":"crmid"},{"VIZVRM6546":"crmid"},{"VIZVRM6547":"crmid"},{"VIZVRM6548":"crmid"},{"VIZVRM6549":"crmid"},{"VIZVRM6550":"crmid"}]'

  4. On successful deployment, validate the deployment as shown below.