How-To: Configure AWS CloudWatch Logs for events to be pulled by Skyformation Cloud Connector

In version 2.3.135 a new feature was introduced that now allows to fetch data from all CloudWatch Logs Groups in a specific AWS region using a single S3 bucket (located in that same region).

The behind the scenes process is that the application:

• Identifies all existing CloudWatch Logs Groups in the account (in a given region)
• The application requests AWS CWL to export small timeframes of data to an S3 bucket
• Once the export process is finished, the application would read it from the bucket
• When it is finished reading the data from the S3 bucket it would delete it from the bucket

Configuration Steps:

Important Note: Following version 2.5.138, CloudWatch logs endpoint has been split to 3 endpoints:

• CloudWatch Logs region log-group (will be referred to as CloudWatch Logs explorer)

• CloudWatch Logs Poller region log-group

• CloudWatch Logs S3 downloader region log-group

To have CC pull data from CloudWatch Logs all 3 endpoints must be active.

To stop retrieval from a specific log-group, it suffices to stop the explorer endpoint.

Note: AWS imposes a strict limit of 1 (one) running export task per region. This means that if you have multiple log-groups in the same region, only one of the endpoints will be able to retrieve data at any given time. This also means that the rest of the explorer endpoints, which do not have an export task in progress will show ERROR followed by a LimitExceededException. CC is configured to retry creating the failed export task until it succeeds (this will happen when the already running export task is finished). Thus, it is advised to stop the explorer endpoints of log-groups you are not interested in getting events from, in order to not have them compete over the AWS API quota.

1. Create an S3 bucket in each region where you would like to pull the CloudWatch logs. The name of the bucket will be used in the next step of the configuration, providing the relevant user the permissions to read this bucket.

2. Allow CloudWatch Logs to export (put objects) into the S3 bucket by setting the permission on the S3 bucket (taken from this guide, step #3)

   Copy the relevant policy JSON (see the 3 options below) into the bucket ACL permissions:

   Navigate to Services -> S3, find your bucket. Click on your bucket, navigate to "permissions"  and then "Bucket Policy". Paste the JSON into the online editor (replace the string "us.west-2" with your relevant region) and save.

• If the bucket is in your account, use the following policy.

{
  "Version": "2012-10-17",
  "Statement": [
   {
     "Action": "s3:GetBucketAcl",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs",
     "Principal": {
        "Service": "logs.us-west-2.amazonaws.com"
     }
   },
   {
     "Action": "s3:PutObject",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs/*",
     "Condition": {
        "StringEquals": {
           "s3:x-amz-acl": "bucket-owner-full-control"
        }
      },
      "Principal": {
        "Service": "logs.us-west-2.amazonaws.com"
      }
    }
  ]
}

• If the bucket is in a different account, use the following policy instead. It includes an additional statement using the IAM user you created in the previous step. 

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Action": "s3:GetBucketAcl",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs",
     "Principal": {
       "Service": "logs.us-west-2.amazonaws.com"
     }
   },
   {
     "Action": "s3:PutObject",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
     "Condition": {
       "StringEquals": {
         "s3:x-amz-acl": "bucket-owner-full-control"
       }
     },
     "Principal": {
       "Service": "logs.us-west-2.amazonaws.com"
     }
   },
   {
     "Action": "s3:PutObject",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
     "Condition": {
        "StringEquals": {
           "s3:x-amz-acl": "bucket-owner-full-control"
        }
     },
     "Principal": {
       "AWS": "arn:aws:iam::SendingAccountID:user/CWLExportUser"
     }
   }
 ]
}

• If the bucket is in a different account, and you are using an IAM role instead of an IAM user, use the following policy instead.  

  {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Action": "s3:GetBucketAcl",
       "Effect": "Allow",
       "Resource": "arn:aws:s3:::my-exported-logs",
       "Principal": {
         "Service": "logs.us-west-2.amazonaws.com"
       }
     },
     {
       "Action": "s3:PutObject",
       "Effect": "Allow",
       "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
       "Condition": {
         "StringEquals": {
             "s3:x-amz-acl": "bucket-owner-full-control"
         }
       },
       "Principal": {
         "Service": "logs.us-west-2.amazonaws.com"
       }
     },
     {
       "Action": "s3:PutObject",
       "Effect": "Allow",
       "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
       "Condition": {
         "StringEquals": {
           "s3:x-amz-acl": "bucket-owner-full-control"
         }
       },
       "Principal": {
         "AWS": "arn:aws:iam::SendingAccountID:role/CWLExportUser"
       }
     }
   ]
  }

1.  

    

   CloudWatch Logs Groups are detected automatically, every 10 minutes.

   Newly discovered groups start automatically to pull data. In order to pause retrieval from CWL Group, click "Edit", and "STOP" on the relevant endpoint.