Deploying CDP DMP SST

This section provides detailed instructions on how to deploy HCL CDP DMP SST using the Devtron in the AWS.

Prerequisites:

Make sure to create dmp-sst secret with required data in the AWS secret manager before deploying CDP DMP SST.

To create the dmp-sst secret in the AWS secret manager, follow the steps below:

  1. Create a dmp-sst secret sample key and value in the dmp-sst secret, and update ConfigMaps data with actual values.
    {
  "AWSRegion": "<AWSRegion>",
  "AerospikeNamespace": "<AerospikeNamespace>",
  "AerospikeServerAddress": "<AerospikeServerAddress>",
  "AerospikeServerPort": "<AerospikeServerPort>",
  "AerospikeTTL": "15552000",
  "AerospikeTimeOut": "10000",
  "AlertEmail": "",
  "AlertFromEmail": "",
  "CookieCacheThreshold": "60",
  "DbDriver": "<DbDriver>",
  "DbHost": "<DbHost>",
  "DbPassword": "<DbPassword>",
  "DbPort": "<DbPort>",
  "DbUrl": "jdbc:mariadb://<ip:port>/dbName",
  "DbUsername": "<DbUsername>",
  "GraphDBEnable": "false",
  "GraphDBEndpoint": "<GraphDBEndpoint>",
  "GraphDBPort": "<GraphDBPort>",
  "HBaseDBEnable": "false",
  "HbaseZookeeperClientPort": "",
  "HbaseZookeeperQuorum": "",
  "KafkaBootstrapServer": "<KafkaBootstrapServer>",
  "KafkaConsumerGroup": "<dmp-sst-kafka-k8>",
  "KafkaPassword": "<KafkaPassword>",
  "KafkaProducerHostName": "<KafkaPassword>",
  "KafkaProducerTopic": "<dmp_sst_rt_profile>",
  "KafkaProfileProducerTopic": "<dmp_sst_profile>",
  "KafkaTopicConsumerBatchSize": "<10,10,10,10>",
  "KafkaTopicConsumerDefaultBatchTimeoutMS": "<5000,5000,5000,5000>",
  "KafkaTopicConsumerDefaultBatchTimeoutTrigger": "<60000,10000,10000,10000>",
  "KafkaTopicConsumers": "<6,6,6,6>",
  "KafkaTopicList": "<dmp_sst_nba,dmp_sst_nba_di_api,analyze_post,dmp_sst_data>",
  "KafkaUsername": "<KafkaUsername>",
  "SstApiAuthKeys": "<HdxDeMMHvUyrqlSBHyaCaQGA>",
  "TsdbHost": "<TsdbHost>",
  "TsdbPort": "<TsdbPort>",
  "ZookeeperClients": "<15>",
  "ZookeeperLockAddress": "<cdp-dmp-sst-zookeeper.cdp-dev-app:2181>",
  "ZookeeperLockAddressOld": "",
  "ZookeeperRetrySleepTime": "<1000>",
  "region": "<region>",
  "type": "<realtime>"
}

Deploying CDP DMP SST

To deploy the CDP DMP SST, follow the steps below:

  1. Navigate to the Devtron Chart Store, and select the cdp-dmp-sst chart to deploy.

  2. Now, configure and deploy the CDP DMP SST charts.

  3. In the YAML section, update the ConfigMap using below details, and deploy the charts.
    SSTExecutorServiceMinThreads: "<SST_EXECUTOR_MIN_THREADS>"
SSTExecutorServiceMaxThreads: "<SST_EXECUTOR_MAX_THREADS>"
SSTExecutorServiceThreadTTL: "<SST_EXECUTOR_THREAD_TTL>"
SSTExecutorServiceQueueSize: "<SST_EXECUTOR_QUEUE_SIZE>"

SRExecutorServiceMinThreads: "<SR_EXECUTOR_MIN_THREADS>"
SRExecutorServiceMaxThreads: "<SR_EXECUTOR_MAX_THREADS>"
SRExecutorServiceThreadTTL: "<SR_EXECUTOR_THREAD_TTL>"
SRExecutorServiceQueueSize: "<SR_EXECUTOR_QUEUE_SIZE>"

ExecutorServiceMinThreads: "<EXECUTOR_MIN_THREADS>"
ExecutorServiceMaxThreads: "<EXECUTOR_MAX_THREADS>"
ExecutorServiceThreadTTL: "<EXECUTOR_THREAD_TTL>"
ExecutorServiceQueueSize: "<EXECUTOR_QUEUE_SIZE>"

MaxEventBatchLag: "<MAX_EVENT_BATCH_LAG>"
SRMaxEventBatchLag: "<SR_MAX_EVENT_BATCH_LAG>"
MaxBatchExecuteRetries: "<MAX_BATCH_EXECUTE_RETRIES>"

KafkaSstEventsTopic: "<KAFKA_SST_EVENTS_TOPIC>"
SECURITY_PROTOCOL: "<SECURITY_PROTOCOL>"
SASL_MECHANISM: "<SASL_MECHANISM>"
KafkaProducerEventType: "<KAFKA_PRODUCER_EVENT_TYPE>"
KafkaProducerPoolSize: "<KAFKA_PRODUCER_POOL_SIZE>"

KafkaDmpSstBidTopic: "<KAFKA_DMP_SST_BID_TOPIC>"
KafkaDmpSstImprTopic: "<KAFKA_DMP_SST_IMPR_TOPIC>"
KafkaDmpSstTaxonomyTopic: "<KAFKA_DMP_SST_TAXONOMY_TOPIC>"
KafkaDmpSegmentsTopic: "<KAFKA_DMP_SEGMENTS_TOPIC>"
KafkaDmpRtSegmentsRefreshTopic: "<KAFKA_DMP_RT_SEGMENTS_REFRESH_TOPIC>"
KafkaDmpSstRtProfileTopic: "<KAFKA_DMP_SST_RT_PROFILE_TOPIC>"
KafkaDmpSstProgImprTopic: "<KAFKA_DMP_SST_PROG_IMPR_TOPIC>"
KafkaDmpSstProgClicksTopic: "<KAFKA_DMP_SST_PROG_CLICKS_TOPIC>"
KafkaDmpSstSiteRecoImprTopic: "<KAFKA_DMP_SST_SITE_RECO_IMPR_TOPIC>"
KafkaDmpSstSiteRecoClicksTopic: "<KAFKA_DMP_SST_SITE_RECO_CLICKS_TOPIC>"
KafkaDmpSstSiteNotifImprTopic: "<KAFKA_DMP_SST_SITE_NOTIF_IMPR_TOPIC>"
KafkaDmpSstSiteNotifClicksTopic: "<KAFKA_DMP_SST_SITE_NOTIF_CLICKS_TOPIC>"
KafkaDmpSstWebPushMetricsTopic: "<KAFKA_DMP_SST_WEBPUSH_METRICS_TOPIC>"
KafkaDmpSstWebPushSntMetricsTopic: "<KAFKA_DMP_SST_WEBPUSH_SENT_METRICS_TOPIC>"
KafkaDmpSstMetricsTopic: "<KAFKA_DMP_SST_METRICS_TOPIC>"
KafkaSMSEmailMetricsTopic: "<KAFKA_SMS_EMAIL_METRICS_TOPIC>"
KafkaWebPushCleanupTopic: "<KAFKA_WEBPUSH_CLEANUP_TOPIC>"
KafkaAppPushCleanupTopic: "<KAFKA_APPPUSH_CLEANUP_TOPIC>"
KafkaGeofenceTopic: "<KAFKA_GEOFENCE_TOPIC>"
KafkaAnalyzePostTopic: "<KAFKA_ANALYZE_POST_TOPIC>"
KafkaWhatsappSubscriptionTopic: "<KAFKA_WHATSAPP_SUBSCRIPTION_TOPIC>"
KafkaDmpSstProfileTopic: "<KAFKA_DMP_SST_PROFILE_TOPIC>"
KafkaSstMergeEventTopic: "<KAFKA_SST_MERGE_EVENT_TOPIC>"
KafkaSstBlacklistEventTopic: "<KAFKA_SST_BLACKLIST_EVENT_TOPIC>"
KafkaDmpSstNbaDiApiTopic: "<KAFKA_DMP_SST_NBA_DI_API_TOPIC>"
KafkaDmpSstDataTopic: "<KAFKA_DMP_SST_DATA_TOPIC>"
KafkaDmpRtSegmentsTopic: "<KAFKA_DMP_RT_SEGMENTS_TOPIC>"
KafkaDmpSstNbaTopic: "<KAFKA_DMP_SST_NBA_TOPIC>"
KafkaSstDemoEventsTopic: "<KAFKA_SST_DEMO_EVENTS_TOPIC>"

DBDataOidsCapLimit: "<DB_DATA_OIDS_CAP_LIMIT>"
MetricLoggingFrequencyMs: "<METRIC_LOGGING_FREQUENCY_MS>"
MaxWaitForConnectionReestablish: "<MAX_WAIT_FOR_CONNECTION_REESTABLISH>"

TSDBHostName: "<TSDB_HOST_NAME>"
GmondCollectors: "<GMOND_COLLECTORS>"
GmondPort: "<GMOND_PORT>"
HostNameOverride: "<HOSTNAME_OVERRIDE>"
MetricPrefix: "<METRIC_PREFIX>"

GeoLiteCityFile: "<GEOLITE_CITY_FILE_PATH>"
GeoISPFile: "<GEOISP_FILE_PATH>"

SSTCronTime: "<SST_CRON_EXPRESSION>"

SSTValidValuesConfig: "<SST_VALID_VALUES_CONFIG_PATH>"
SSTValuesMappingConfig: "<SST_VALUES_MAPPING_CONFIG_PATH>"

ValidValuesS3Path: "<VALID_VALUES_S3_PATH>"
ValuesMappingS3Path: "<VALUES_MAPPING_S3_PATH>"

EncryptionEnabled: "<BOOLEAN>"

fromEmailId: "<FROM_EMAIL_ID>"
toEmailId: "<TO_EMAIL_ID>"

DemoCampaignList: "<DEMO_CAMPAIGN_LIST>"
CampaignIncludeList: "<CAMPAIGN_INCLUDE_LIST>"

Dbname: "<DATABASE_NAME>"

mapsGenLocation: "<MAPS_GEN_LOCATION>"
ValidValuesConfig: "<VALID_VALUES_CONFIG_PATH>"
ValuesMappingConfig: "<VALUES_MAPPING_CONFIG_PATH>"

mapsCronTime: "<MAPS_CRON_EXPRESSION>"
deleteOlderBackupIntervalInDays: "<BACKUP_RETENTION_DAYS>"

KMS_IMPLEMENTATION: "<KMS_IMPLEMENTATION>"

LOG_LEVEL_APP: "<LOG_LEVEL_APP>"
LOG_LEVEL_ROOT: "<LOG_LEVEL_ROOT>"

ArrayDefaultLimit: "<ARRAY_DEFAULT_LIMIT>"
ArrayMaxLimit: "<ARRAY_MAX_LIMIT>"

  4. On successful deployment, validate the deployment as shown below.