Amazon SQS Adapter

Overview

This document provides an overview and the functional specifications of the Amazon SQS Adapter. The Amazon Simple Queue Service (SQS) is a distributed message-queuing service supporting programmatic sending of messages via web service applications. This message-queuing service is entirely manageable and scalable and can dissociate microservices, distributed systems, and server-less applications. The function of an Amazon SQS adapter is to access queues located at various Amazon Web Services-served geographic regions to fetch messages from standard queues and also store messages in standard queues in Amazon SQS.

IAM users can access SQS messages if they have the correct AWS access key and secret key, and also the queue-access permission. The SQS Adapter supports the sending and fetching of message attributes along with the message texts. SQS adapter supports standard queues having at-least-once delivery. For more information, see https://aws.amazon.com/sqs/.

The Amazon SQS adapter does not support the predefined-schema-import option. An AWS Identity and Access Management (IAM) user can be created in AWS for accessing the SQS services. This user represents a person or application that will interact with AWS. It is not the same as the AWS account root user. For more information, see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html.

Authenticating connections

The Amazon SQS adapter uses AWS SDK programmatically, to verify the user identity in programmatic calls, the user needs to provide AWS access keys. Typically, access keys are either temporary (short-term) credentials or long-term credentials, such as those for IAM users or the root user of an AWS account. In many cases, there are alternatives to long-term keys to consider.

Amazon SQS adapter as a source

Amazon SQS adapter as a source supports:

Read message from the queue.
Read message from the queue and delete after reading.

Amazon SQS adapter as a target

Amazon SQS adapter as a target supports:

Send message to the existing queue.
Send message to the queue and create the queue, if does not exist.

Adapter properties and commands

This section provides the details of the adapter properties to be set to define connections and actions and import schemas in the new web UI of HCL Link.

Access key

Specifies the access key of AWS user for the connection. This is a mandatory property. The access key is used to sign programmatic requests to AWS API calls through the adapter. The corresponding adapter command is -AK access_key (or -ACCESSKEY access_key).

Secret key

Specifies the secret key of AWS user for the connection. For the username and password, both the access key ID and secret access key are mandatory to authenticate SQS queue GET or PUT request. The corresponding adapter command is -SK secret_key (or -SECRETKEY secret_key).

Region name

Specifies the region name to connect to an AWS region for the IAM user account. This is a mandatory property. The corresponding adapter command is -R region_name (or -REGION region_name).

Queue

Specifies the SQS queue name to or from which messages need to be accessed. This is a mandatory property. During action configuration, SQS adapter provides options to select queue name among available queues in the provided region. The corresponding adapter command is -Q queue_name (or -QUEUE queue_name).

Create queue

If specified, this property indicates the SQS adapter to create a queue with the name passed in the ‘queue_name’ property in case it does not exist. This property is optional. When the CreateQueue property is not specified, adapter checks if the queue exists; if not, it throws an error. The corresponding adapter command is -CQ (or -CREATEQUEUE).

Attribute

Specifies if the SQS message should be accessed along with the corresponding message attributes. When the attributes flag is set to true in output card, the passed data should be in predefined JSON format, else data should be in a plain text message.

{"message": "text message", "attributes": [{"name": "attribute_name", "type= "String", "value": "attribute_value"}]

When the attributes property is present in input card, SQS adapters get the data in the above JSON format; otherwise, the text message is fetched from SQS queue. If it is omitted, attributes are not used. The corresponding adapter command is -A (Or -ATTRIBUTES). For more information on Amazon SQS message attributes, see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-attributes.html.

Timeout

Specifies the time duration (in seconds) to wait for before retrieving the message from the queue. The timeout property of SQS adapter is applicable and utilized only in HCL Link deployments. The SQS adapter would listen to the queue for specified seconds. The corresponding adapter command is -LSN name.

Limit

Specifies the total number of messages to consume. Minimum value should be 1. The value S means all available messages. If no limit is specified, adapter fetches only one message at a time from the queue. The limit property is applicable and utilized only in HCL Link deployments. The corresponding adapter command is -QTY limit.

Logging

Specifies the level of logging to use for the log (trace) file produced by the adapter. The default is Off. The value Information means log informational, the value Errors Only means log error messages only, and the value Verbose means log debug and trace level messages along with the informational and error messages.

The corresponding adapter command is:

-T [E|V] [+] [file_path]

-T -> Log adapter informational messages.

-TE -> Log only adapter errors.

-TV -> Use verbose (debug) logging. The log file records all activity that occurs while the adapter is producing or consuming messages.

+ -> Appends the trace information to the existing log file. Omit this argument to create a new log file.

file_name -> Specifies the location of the log file to which to write log messages. If not specified, the adapter creates the m4sqs.trc log file in the map directory.

Append log

Flag indicating what to do if the specified log file already exists. When set to true, the log messages are appended to the file. When set to false, the file is truncated, and the messages are written to the empty file. The default value is true.

Log file name

Specifies the location of the log file to which to write log messages. If not specified, the default log file name m4sqs.trc is used, and the file is stored to the directory in which the executed compiled map resides.

Examples

Examples of the GET map function

In the following example, assume that the adapter being used in a GET function is defined as follows:

```
GET("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test A", inputdata)
```
Since the -A/-ATTRIBUTES command is not present, the adapter only text message from the queue.

Assume the GET map function defined as follows:

GET("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test -A", inputdata)

In this case the adapter retrieves message attributes along with the text message and the output data defined as follows:
```
{"messages": "It's get example", "attributes": [{"name": "attribute1", "type"= "String", "value": "attribute example"}]}
```

Examples of the PUT map function

In the following example, assume that the adapter being used in a PUT function defined as follows:

Example of create queue property use in PUT map function:
```
PUT("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test", inputdata)
```
In this case if the ‘queue-for-test’ does not exist in specified region, the adapter will show error, since the -CQ/-CREATEQUEUE is not present.
Assume the PUT map function defined as follows:
```
PUT("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test -CQ", inputdata)
```
In this case the adapter creates the queue and send message to the newly created queue.
Example of attribute property use in PUT map function:
```
PUT("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test", inputdata)
```
Since the -A/-ATTRIBUTES command is not present, input data is treated as message body to be sent over SQS queue

Assume the PUT map function defined as follows:

PUT("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test -A", inputdata)

In this case the adapter assumes the input data is passed in required JSON format; input data defined as follows:
```
{
   "message":"It’s put example",
   "attributes": [
        {"name":"attribute1", "type":"String", "value":"attribute example"}        
   ]
}
```