- Introduction
- Prerequisites
Before installation, ensure AWS account access with necessary permissions, VPC setup, subnets, KMS key, and installation of CLI tools like AWS CLI, kubectl, and Helm
- Installing Devtron
Devtron is installed using Helm to manage the Kubernetes-based CDP microservices deployment.
- Installing Aerospike
This section provides a step-by-step guide to installing Aerospike.
- Installing MongoDB
MongoDB is configured as a replica set across multiple nodes with an arbiter, and includes user and role setup. Monitoring is enabled using MongoDB Exporter.
- Installing Druid
Apache Druid is deployed for real-time analytics, using MySQL for metadata and S3 for deep storage. Configuration includes connecting storage and starting Druid services.
- Installing MSK
AWS Managed Streaming for Kafka (MSK) is configured with IAM-based and SASL authentication. Secrets are managed in AWS Secrets Manager and linked to MSK.
- Installing EMR
Amazon Elastic MapReduce (EMR) is set up with Hadoop and HBase to handle distributed data processing. Logs and data are stored in S3 buckets.
- Installing MWAA
Amazon Managed Workflows for Apache Airflow (MWAA) provides Airflow orchestration for CDP jobs. The environment is configured with custom DAGs and logging enabled.
- Installing Postgres
PostgreSQL is deployed with master-slave replication using Docker Compose. Schema setup includes essential tables and indexes.
- Configuring AWS KMS
AWS Key Management Service (KMS) is used to create and manage encryption keys. These are used with Secrets Manager and other encrypted services.
- Configuring AWS Secrets
AWS Secrets Manager stores sensitive credentials like DB passwords and Kafka tokens, encrypted with KMS.
- CDP Monitoring Applications Configuration
Monitoring includes Prometheus, Grafana, Node Exporter, Mongo Exporter, and Fluentbit for log aggregation.
