In this guide we will configure 2 main themes

1. A tightly secured integration between the S3 bucket where the data we want to retrieve is stored, to a SQS queue, which will be notified when new objects are added to the S3 Bucket
2. A preferred authentication method that'll allow the Custom Application Connector to read the messages in the SQS queue and S3 bucket, with minimal permissions.

Prerequisites

This guide assume an existence of S3 bucket, and a SQS queue.
You should have both the S3 Bucket and the SQS queue's ARNs. They can be obtained via the AWS Web Console, the CLI as well as other methods.

We're going to configure them such that whenever a new object is put into the S3 bucket, the SQS queue will get a notification of that.

In the Custom Application Connector we'll read events off the SQS queue and read the S3 Objects they point to, in order to ingest its data.

Configure S3/SQS integration

Note: If the data is not stored in S3, but instead sent directly to SQS by a third party, skip these steps

In AWS's console, navigate to the S3 bucket where the data it at.

Under "Properties" - 

   Enable event notification to SQS queue in the S3 bucket you would like to get logs from.
        https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-event-notifications.html

        Choose ObjectCreate (all) events to be notified as in the below example.

Allow the S3 Bucket to send events to the SQS Queue

Replace the access policy attached to the queue with the following policy (in the SQS console, you select the queue, and in the Permissions tab, click Edit Policy Document (Advanced). 

{
 "Version": "2012-10-17",
 "Id": "example-ID",
 "Statement": [
  {
   "Sid": "example-statement-ID",
   "Effect": "Allow",
   "Principal": {
    "AWS":"*"  
   },
   "Action": [
    "SQS:SendMessage"
   ],
   "Resource": "SQS-queue-ARN", 
 "Condition": { 
 "ArnLike": { 
 "aws:SourceArn": "arn:aws:s3:*:*:bucket-name" 
 } 
 } 
 } 
 ] 
}

Obtain the authentication properties

        - SQS URL
           Log into your AWS account, go to the SQS service and focus on the SQS Queue used
           for the S3 events notifications. In the Details tab copy the URL value (see example below):

        - SQS Region
           In the same page as above (SQS detail tab) you will see the SQS region. It is the string
          between the sqs and amazonaws strings in the URL.

          At the example above the region is: us-east-1

        - SQS-Message-Origin

Based on which integration you're configuring, messages are put in the queue in different ways.

For messages put in the queue by the above setup, i.e. S3 notifies the SQS directly, select "S3_TO_SQS".

For messages put in the queue by the above method, with that variation that the messages were sent from S3 to SNS, and then configured in SNS to be sent to SQS, select "S3_TO_SNS_TO_SQS".

For messages NOT configured by the above method, that are not pointing to any objects in S3 at all, and are themselves the data that needs to be consumes (usually used by 3rd parties to notify on events directly), select "DIRECT_SQS"

Authentication

There are 3 method for authenticating:

1. InstanceProfile - If the Cloud Connectors is deployed in AWS EC2, this is the easiest method to go
2. STSAssumeRole - AWS's most recommended method of authentication. Also allows to monitor S3/SQS in different AWS accounts
3. IAM User - for legacy reasons appears in the UI as "s3-sqs-authn-type" 

For any of the above methods, an IAM Policy is needed, with minimal permissions that'll allow it to manipulate the SQS queue, and read the objects from the S3 bucket.

Create an IAM Policy

Navigate to IAM service -> Policies

Create a policy with the below document. modify where needed in bold.

{ 
"Version": "2012-10-17", 
"Statement": [ 
    { 
    "Action": [ 
        "sqs:DeleteMessage", 
        "sqs:DeleteMessageBatch", 
        "sqs:ReceiveMessage" 
    ], 
    "Effect": "Allow", 
    "Resource": "Queue ARN" 
    }, 
    { 
    "Action": [ 
        "s3:GetObject" 
    ], 
    "Effect": "Allow", 
    "Resource": "Bucket ARN/*" 
    } 
  ] 
} 

Per your preference of authentication method, follow one of the below sections

Authenticate with InstanceProfile

1. Follow the below steps to create an IAM Role
2. Navigate to IAM -> Roles and search for the role created earlier
3. Allow EC2 instances to use (assume) the role
   1. Go to "Trust relationships" tab, click "Edit trust relationship"
   2. Make sure the "Statements" array contains the document

      {
        "Effect": "Allow",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
       }

   3. Click "Update Trust Policy"
4. Navigate to AWS EC2 service
5. Right-click on the instance where Cloud Connectors in installed, Click Instance Settings -> Attach/Replace IAM Role
6. Choose the role from the drop-down list, and Apply
7. When on-boarding the Custom Application Connector, choose InstanceProfile

Authenticate with IAM User

1. Navigate to IAM -> Users
2. Click "Add user"
3. Set a unique username. e.g. cloud-connectors-customer-connector
4. Check the "Programmatic access" checkbox, clikc "Next: Permissions"
5. Click the box "Attach Existing Policies directly"
6. Search for the policy created earlier, and check its checkbox, click "Next: Tags"
7. Optionally add tags
8. Click "Next: Review"
9. Review and Click "Create user"
10. Copy the Access Key and Secret access key. Optionally download the credentials in csv, as they will not be accessible after you leave this page
11. When on-boarding the Custom Application Connector, choose "s3-sqs-authn-type"

Authenticate with STSAssumeRole

1. Follow the below steps to create an IAM Role
2. If you don't already have an IAM user you wish to use, create an IAM user with programatic access that will be used to use (assume) the role
   1. Follow the steps of "Authenticate with IAM User", except that don't attach any policy to that user (skip steps 5-6). 
      By doing this, this user does not have ANY permissions yet. later it'll only be allowed to "assume the role" with the policy we create in step 1
   2. Keep the user's Access key and Secret Access key
3. Navigate to IAM -> Roles and search for the role created earlier
4. Allow a specific user to use (assume) the role
   1. Go to "Trust relationships" tab, click "Edit trust relationship"
   2. Make sure the "Statements" array contains the document. modify accordingly

      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::123456789012:user/ccuser"
        },
        "Action": "sts:AssumeRole"
       }

   3. Click "Update Trust Policy"
5. Keep the role's ARN
6. When on-boarding the Custom Application Connector, choose "STSAssumeRole"
7. Fill in the credentials
   1. Under Role-Session-Name give a meaningful name to identify sessions initiated by CC, e.g. cc-session
   2. Under Access-Key fill in the IAM User's Access key
   3. Under Secret-Key fill in the IAM User's Secret Access key

Create IAM Role

1. Navigate to IAM -> Roles. Click "Create Role"
2. Select AWS Service box, and EC2 service and Click "Next: Permissions"
   1. This will create a "Trust relationship" that'll allow Allows EC2 instances to call AWS services on your behalf. We will modify the Trust relationship document later, per authentication method you'll choose
3. Search the policy created earlier and check its checkbox. Click "Next: Tags"
4. Optionally add tags. Click "Next: Review"
5. Give it a meaningful name, e.g. CCCustomConnectorRole, and description. Click "Create Role"
6. Search for the created role and click on it. Copy aside the role's ARN