Deploying CDP DMP Producer

This section provides detailed instructions on how to deploy HCL CDP DMP Producer using the Devtron in the OpenShift.

Prerequisites:

Make sure to create a cdp-producer secret with required data in the HashiCorp Vault before deploying CDP DMP Producer.

To create the cdp-producer secret in the HashiCorp Vault, follow the steps below:

Create a cdp-producer secret sample key and value in the UI secret, and update ConfigMaps data with actual values.

{
  "DbPassword": "",
  "DbUrl": "jdbc:mariadb://:3306/vrm?useSSL:false",
  "DbUsername": "",
  "KafkaProducerConfig": "",
  "SMTPhost": "",
  "SMTPpassword": "",
  "SMTPport": "20225",
  "TsdbHost": "",
  "TsdbPort": "443",
  "kmsdbpassword": "",
  "kmsdbsername": "",
  "kmsdburl": "jdbc:mariadb://:3306/vrm?useSSL:false",
  "piidbpassword": "",
  "piidbsername": "",
  "piidburl": "jdbc:mariadb://:3306/vrm?useSSL:false"
}

Create a Kubernetes Secret with Vault credentials (username and password) to fetch the secrets from vault as shown below.
```
apiVersion: v1
kind: Secret
metadata:
  name: vault-creds
  namespace: cdp-dev-app
type: Opaque
data:
  username: <vault username>
  password: <vault password>
```
Create required kafka topics before deployment. Sample Kafka topic configuration is shown below.

Deploying CDP DMP Producer

To deploy the CDP DMP Producer, follow the steps below:

Navigate to the Devtron Chart Store, and select the cdp-dmp-producer chart to deploy.
Now, configure and deploy the CDP DMP Producer charts.

In the YAML section, update the ConfigMap using below details, and deploy the charts.

BucketType: minio
ConfigRefreshInterval: "10"
CronExpr: 0 */5 * * * ?
DMPProducerBatchSize: "1000"
DMPProducerEventType: "100"
DbDriver: org.mariadb.jdbc.Driver
EncryptionEnabled: "true"
EventBatchSize: "1000"
EventQueueLength: "100000"
EventThreadPoolMax: "96"
EventThreadPoolMin: "96"
EventThreadTTL: "5"
GraphEnabledCampaignIds: dummy
HostNameOverride: dmp-sst-producer-mu
IS_SMTP: "true"
KMS_HASHICORPVAULT_ENDPOINT: ""
KMS_HASHICORPVAULT_ENDPOINT_TOKEN: ""
KMS_IMPLEMENTATION: HashiCorpVault
KafkaExecutorServiceMaxThreads: "32"
KafkaExecutorServiceMinThreads: "32"
KafkaExecutorServiceQueuseSize: "1"
KafkaExecutorServiceThreadTTL: "5"
KafkaProducerEventType: "1111"
KafkaProducerGraphNbaTopicName: dmp_sst_nba_graph
KafkaProducerHostName: dmp-sst-producer-mu
KafkaProducerHostname: dmp-producer-hcl
KafkaProducerMeta: "true"
KafkaProducerNbaTopicName: dmp_sst_nba
KafkaProducerPoolSize: "10"
KafkaProducerTaxonomyTopicName: dmp_sst_taxonomy
LOG_LEVEL_APP: info
LOG_LEVEL_ROOT: info
MaxEventBatchLag: "16"
MaxS3DownloadRetries: "3"
MaxWaitForConnectionReestablish: "5"
MetricPrefix: dmp-sst-producer-mu
MinioAccessKey: ""
MinioEndPointUrl: ""
MinioSecret: ""
NbaFileBatchSleepInMs: "1000"
RawNBAConfigS3Location: ""/sst/config
RawNbaConfigDownloadDir: /disk1/sst/nba/config
RawNbaDownloadDir: /disk1/sst/nba/data
SMTPhost: ""
SMTPpassword: ""
SMTPport: ""
SMTPusername: ""
SchedulerIntervalInMin: "2"
SegmentNameMacro: SegmentNameMacro
TSDBHostName: dmp-sst-producer-mu
TSDBServerAddress: ""
TSDBServerPort: "443"
TaxonomyDownloadDir: /disk1/sst/taxonomy
alertToEmailId: ""
allowedRegions: aps1
aps1_host: ""
aps1_namespace: cdpstore
aps1_port: "13000"
defaultSegmentNotifyEmail: ""
emailType: SMTP
expiry: "-1"
failureSubject: Ingestion Failed
fromEmailId: ""
funnelSegmentNameMacro: |
  "{ segmentname }": null
funnelWaitTime: "60000"
hashicorpvault.endpoint: ""
hashicorpvault.endpoint.token: ""
host: ""
kms.implementation: HashiCorpVault
namespace: cdpstore
pii_expiry: "-1"
pii_host: ""
pii_namespace: cdpstore
pii_port: "13000"
pii_region: aps1
port: "13000"
queue_names: TEST
queue_provider: activemq
region: aps1
segmentIdMacro: segmentIdentifier
sqs_threads: "12"
sqs_url_prefix: ""
successSubject: Ingestion Completed
supported_regions: aps1
timeout: "3000"
uploadSuccessEmailIds: ""
userListFailedHTML: /mnt/sst/UserListFailed.html
userListInternalErrorHTML: /mnt/sst/UserListInternalError.html
userUploadErrorFunnelHTML: /mnt/sst/Failed.html
userUploadFailureFunnelHTML: /mnt/sst/Failed.html
userUploadSuccessFunnelHTML: /mnt/sst/success.html
userUploadSuccessHTML: /mnt/sst/UserUploadSuccess.html

On successful deployment, validate the deployment as shown below.