Adapter Properties and Commands
This chapter provides a detailed description of the adapter.
Access Key
Specifies the key of AWS user for the connection. This access key is used to sign programmatic requests to AWS API calls through the adapter. The corresponding adapter command is -AK (or -ACCESSKEY).
Secret Key
Specifies the key of AWS user for the connection. Like the username and password, both the access key ID and secret access key are mandatory to authenticate S3 object for a Get or Put request. The corresponding adapter command is -SK (or -SECRETKEY).
Region Name
Specifies the name to connect to an AWS region for the IAM user account. The corresponding adapter command is -R (or -REGION).
Bucket Name
Specifies the name in which an S3 object would reside. During the action configuration, S3 Adapter provides options to select bucket name among available buckets. The corresponding adapter command is -B (or -BUCKET).
Folder Name
Specifies the folder name where S3 object may reside inside a bucket. This property is optional because an object may directly reside inside a bucket or under folder. When the folder property is not specified, adapter takes the direct path of the bucket to access the object. Otherwise, the folder name is prefixed to the key name. The corresponding adapter command is -F (or -FOLDER).
Object Name
Specifies the name of the S3 object to be accessed. A key can reside directly in a bucket or a folder inside the bucket. The corresponding adapter command is -K (or -KEY).
Content Type
Specifies the field that indicates the media type of the entity-body sent to Amazon S3. If the type is not set, adapter takes the default type, which is a standard MIME-type that describes the format of the content. For more information, go to https://www.w3.org/Protocols/rfc1341/4_Content-Type.html.
Type: String
Default: binary/octet-stream
Valid Values: MIME types
Constraints: None
The corresponding adapter command is -CT (or -CONTENTTYPE).
Content Language
Sets the Content-Language HTTP header, which specifies the natural language(s) of the intended audience for the enclosed entity. If no language is set, S3 adapter takes the default value ‘en’. The corresponding adapter command is -CL (or -CONTENTLANG).
Encryption
Specifies the server-side encryption algorithm while encrypting the object using the AWS-managed keys. When the property is not specified, data uploaded to S3 remains unencrypted. The corresponding adapter command is -E (or -ENCRYPTION).
Valid Values: AES256, KMS.
AES256: 256 bit – Advanced Encryption Standard is a data or file encryption technique. The result of the process is downloadable in a text file. AES performs all its computations on bytes rather than bits. Hence, AES treats the 256 bits of a plaintext block as 16 bytes. It uses 14 rounds for 256-bit keys. For more information, see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html.
KMS: It represents the AWS Key Management Service (AWS KMS). The master keys that the user creates in AWS KMS are used to encrypt or decrypt the object uploaded to S3. For more information on KMS encryption technique, see https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html.
Logging
Specifies the level of logging to use the log (trace) file produced by the adapter. The default is Off. The value Information means log informational, the value Errors Only means log error messages only, and the value Verbose means log debug and trace level messages along with the informational and error messages.
Logging can be specified using command:
-T [E|V] [+] [file_path]
-T Log adapter informational messages.
-TE Log only adapter errors during map execution.
-TV Use verbose (debug) logging. The log file records all activity that occurs while the adapter is producing or consuming messages.
+ Appends the trace information to the existing log file. Omit this keyword to create a new log file.
file_path The full path to the adapter trace log. If you omit this keyword, the adapter creates the m4azblob.mtr log file in the map directory.
Append Log
Specifies the flag that indicates the action to take when the specified log file already exists. When set to true, the log messages are appended to the file. When set to false, the file is truncated, and the messages are written to the empty file. The default value is true.
Read Mode
- List: To list all the objects from the S3 bucket
- Keep: To keep an object in the S3 bucket
- Delete: To delete an object after reading the data
The corresponding adapter commands is -RM (or -READMODE).
Wildcard
If the Read mode is List, lists all the objects in the S3 bucket. To filter desired objects from the listed objects in the bucket, specify Key prefix and/or Key delimiter options. Prefixes and delimiters are used to organize the objects in the S3 bucket.
The corresponding adapter commands are -KP (or -KEYPREFIX) and -KD (or -KEYDELIMITER).
Endpoint
Specifies the endpoint to connect and access the S3 or S3 compatible buckets by the Amazon S3 adapter. If not specified, Amazon S3 adapter connects to the Amazon S3 Service endpoint based on the region specified.
To access an S3 compatible bucket hosted by an object storage service other than the default Amazon S3 service, Endpoint option must be specified. Region option is optional. The corresponding adapter command is -EP (or -ENDPOINT).
Log File Path
Specifies the location of the log file to which the log messages are written. If not specified, the default log file name m4s3.trc is used, and the file is stored to the directory in which the executed compiled map resides.
Convert Data
Specifies to convert data to parsed form. if enabled, support for converting CSV, Avro, Excel, Parquet and JSON documents into sets of data records in the CSV format. This means that the data produced by the converters can be loaded directly into relational databases.
Data Format
- CSV
- Excel
- Avro
- Parquet
- JSON
Header
If enabled, specifies whether the CSV data contains a header row.
Quote
Specifies the quote character. Default value is ".
Delimiter
Specifies the field delimiter character. Default value is ,.
Escape
Specifies the escape character.
Separator
Specifies the row separator characters. Default value is \r\n.
Character Set
Specifies the character set of the data. Default value is UTF-8.
Limit
Specifies the maximum number of records to return.
Sample Size
Specifies the number of records to analyze when determining the data structure. Default value is 100.
Worksheet Index
Specifies the index of the worksheet. The index of the first worksheet is 1 (default).
Select Source Tables
Select which arrays of objects in the AVRO data you wish to map to tables in the target.
Find Array
If enabled, specifies that the output will iterate on the first array found in the JSON document.
Array Path
Specifies the path of the JSON array within the document. Omit if the document is an array. For example,"/resources".