Amazon S3 Connector

Introduction

The Amazon S3 adapter and Amazon S3 Connector retrieve user behavior data from multiple files and send them to HCL CDP. The connector supports sending data from Parquet, CSV, and JSON files to efficiently integrate with HCL CDP.

Prerequisites

Before using the Amazon S3 adapter and Amazon S3 Connector, ensure you have the following details:

Amazon S3 Service User Account
Amazon S3 Account Access Key
Amazon S3 Account Secret Key
Amazon account region details
Amazon S3 ARN role details

Connection Properties

The CDP administrator must configure the following connection properties:


Property	Description
Amazon S3 Account Name	The name of the Amazon S3 service user account where files are stored.
Amazon S3 Access Key	Amazon S3 Access Key for authentication.
Amazon S3 Secret Key	Amazon S3 Secret Key for authentication.
Amazon S3 ARN role	Amazon S3 ARN role for authentication.
Amazon S3 Region	Amazon S3 region for authentication.
Amazon S3 Duration	Time period for which temporary credentials remain valid.
CDP API URL	The CDP API endpoint URL.
CDP API Key	API key used to authenticate the CDP tenant.
CDP API Pass Key	API pass key used to authenticate the CDP tenant.
Activity Fetch Units	The fetch interval unit (hours, minutes, seconds).
Activity Fetch Frequency	The fetch interval time (e.g., 2).
Number of Retries	The number of retry attempts for the API call.

Action Properties

Define the following properties when configuring an Amazon S3 connector from an action box:


Property	Description
Amazon S3 Bucket Name	Name of the Amazon S3 bucket where the files are located.
Amazon S3 Folder Name	Name of the Amazon S3 Folder where the files are located.
File Type	The file format: csv, json, or parquet.
User ID Field Name	Name of the user ID field in the input file (Mandatory).
EventName Field Name	Name of the event name field in the input file (Mandatory).
CDP Primary Identifier	Primary Identifier name for HCL CDP (e.g., identity).
Duration (in days)	Number of days the flow should run.
Maximum Instances	Number of instances to run in parallel. Default: 2.
Batch Size	Batch size to send to HCL CDP. Default: 500.
results_dir	Set to `/data/results`.
run_identifier	Set value to 1.

Folder Properties

For HCL Link (Cloud Native / Kubernetes):

Create aws_watcher_s3, zipped_files and results folders in the /data directory. Use chmod 777 -R permissions.

For HCL Link (Docker):

Create the same folders in the /opt/hip-rest/ directory with 777 -R permissions.

Compression Requirements

Before uploading the CSV, JSON and Parquet files into S3 bucket folder, compress the files as follow:

For CSV and JSON files, compress the files in .gz format. Zip the files using git bash using the below command.
gzip <filename>
For Parquet file, compress the file in Snappy compression.

Limitations

The system supports processing up to 100,000 records per file for CSV, JSON, and Parquet formats. Multiple data arrays are not supported in JSON; only one single data array should be present per file.

JSON Layout and Response Examples

Sample JSON Layout:

{
                "data": [
                {
                "customer_id": 1,
                "name": "John",
                "city": "Mumbai",
                "aws_eventname": "login"
                }
                ]
                }

customer_id and aws_eventname are mandatory fields in the file as header to get user behavior data.

Success Scenario:

"status_message","statusCode","message","id","timestamp"
                "Success","200","Data Processed successfully","ebc2bedf-5535-4a1f-adaf-ee60d0ff3790","2025-04-03T12:20:56"

Error Scenario:

"status_message","statusCode","message","id","timestamp"
"Error","500","Internal server error",,"2025-09-17 13:24:46"

Known Issues and Resolutions


Known Issue	Resolution
FILE NOT EXISTS (Source not available)	Check the Amazon S3 file name, bucket name, and folder name to provide the correct location.
500 / 503 Server Errors (Server busy / Internal error)	Retry after a delay and implement retry logic.