Google Cloud Storage

This guide provides instructions on how to configure and use the Google Cloud Storage (Event Data) Connector to synchronize data between Google Cloud Storage (GCS) and HCL CDP.

Introduction

The Google Cloud Storage (Event Data) Connector facilitates data synchronization between Google Cloud Storage (GCS) and HCL CDP. This integration allows event data files stored in GCS buckets and folders to be fetched, processed, and inserted or updated in HCL CDP.

Prerequisites

Before using the GCS adapter and connector, ensure you have the following:

Google Cloud Storage service account.
Google Cloud Storage service account credentials file.
A directory named gcs_files_watcher created in the /data/ directory.

Connection Properties

To use the GCS connector, the HCL CDP administrator must configure a connection. Establishing communication requires the following parameters:


Property	Description
Service Filename	The proper file name along with its extension. The Service key file must be placed inside the staging directory `/data/tmp`.
CDP API URL	CDP DI API endpoint URL (e.g., https://<base_url>/v3/data).
CDP API Key	CDP DI API key for authentication (x-api-key).
CDP API Pass key	CDP DI API passkey for additional authentication (x-api-passkey).

Action Configurations

When configuring the GCS connector from a Touchpoint or Process Box, define the following properties:


Property	Description
Bucket Name	The bucket name of the Google Cloud Storage (Mandatory).
Folder Name	The folder name of the Google Cloud Storage (Mandatory).
Sub Folder Name	The subfolder where the files are present. Enter the prefix after the date folder.
Primary id	The exact column header in your CSV file to use as the primary identifier.
Event Name	The exact column header in your CSV file to use as the event name.
Duration	The duration for scheduling purposes.

Limitations

The system currently supports processing up to 100,000 records per file for CSV formats. It is recommended to split larger datasets into multiple files to ensure successful processing.

Examples and Event Tracking

The response tracking from the GCS connector contains the fields: status_code, message, and id.

Success Scenario: Report Generation


                This is a Google_Cloud_storage connector run report generated on 2025-12-16T04:38:01
                -----------------------------------------------------------------------------------------
                start time: 2025-12-16T04:38:01
                Total No of Rows :5
                Records updated in CDP:5
                Records failed in CDP:0
                Process ended at: 2025-12-16T04:38:01

Error Scenario Example:


                "status_message","statusCode","message","id","timestamp"
                "Error","500","Internal server error",,"2025-09-17 13:24:46"

Known Issues and Resolutions


Known Issue	Resolution
Error: Source not available (File or folder not found).	Check the folder name and bucket and provide the prefix after the folder name.