How-To: Configure AWS CloudWatch Logs for events to be pulled by Skyformation Cloud Connector

In version 2.3.135 a new feature was introduced that now allows to fetch data from all CloudWatch Logs Groups in a specific AWS region using a single S3 bucket (located in that same region).

The behind the scenes process is that the application:

• Identifies all existing CloudWatch Logs Groups in the account (in a given region)
• The application requests AWS CWL to export small timeframes of data to an S3 bucket
• Once the export process is finished, the application would read it from the bucket
• When it is finished reading the data from the S3 bucket it would delete it from the bucket

Configuration:

1. Create S3 bucket in each region where you would like to pull the CloudWatch logs. The name of the bucket will be used in the next step of the configuration, providing relevant user the permissions to read this bucket.

2. Allow CloudWatch Logs to export (put objects) into the S3 bucket by setting the permission on the S3 bucket (taken from this guide, step #3)

   Copy the relevant policy JSON (see the 3 options below) into the bucket ACL permissions:

   Navigate to Services -> S3, find your bucket. Click on your bucket, navigate to "permissions"  and then "Bucket Policy". Paste the JSON into the online editor (replace the string "us.west-2" with your relevant region) and save.

• If the bucket is in your account, use the following policy.

{
  "Version": "2012-10-17",
  "Statement": [
   {
     "Action": "s3:GetBucketAcl",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs",
     "Principal": {
        "Service": "logs.us-west-2.amazonaws.com"
     }
   },
   {
     "Action": "s3:PutObject",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs/*",
     "Condition": {
        "StringEquals": {
           "s3:x-amz-acl": "bucket-owner-full-control"
        }
      },
      "Principal": {
        "Service": "logs.us-west-2.amazonaws.com"
      }
    }
  ]
}

• If the bucket is in a different account, use the following policy instead. It includes an additional statement using the IAM user you created in the previous step. 

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Action": "s3:GetBucketAcl",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs",
     "Principal": {
       "Service": "logs.us-west-2.amazonaws.com"
     }
   },
   {
     "Action": "s3:PutObject",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
     "Condition": {
       "StringEquals": {
         "s3:x-amz-acl": "bucket-owner-full-control"
       }
     },
     "Principal": {
       "Service": "logs.us-west-2.amazonaws.com"
     }
   },
   {
     "Action": "s3:PutObject",
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
     "Condition": {
        "StringEquals": {
           "s3:x-amz-acl": "bucket-owner-full-control"
        }
     },
     "Principal": {
       "AWS": "arn:aws:iam::SendingAccountID:user/CWLExportUser"
     }
   }
 ]
}

• If the bucket is in a different account, and you are using an IAM role instead of an IAM user, use the following policy instead.  

  {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Action": "s3:GetBucketAcl",
       "Effect": "Allow",
       "Resource": "arn:aws:s3:::my-exported-logs",
       "Principal": {
         "Service": "logs.us-west-2.amazonaws.com"
       }
     },
     {
       "Action": "s3:PutObject",
       "Effect": "Allow",
       "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
       "Condition": {
         "StringEquals": {
             "s3:x-amz-acl": "bucket-owner-full-control"
         }
       },
       "Principal": {
         "Service": "logs.us-west-2.amazonaws.com"
       }
     },
     {
       "Action": "s3:PutObject",
       "Effect": "Allow",
       "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
       "Condition": {
         "StringEquals": {
           "s3:x-amz-acl": "bucket-owner-full-control"
         }
       },
       "Principal": {
         "AWS": "arn:aws:iam::SendingAccountID:role/CWLExportUser"
       }
     }
   ]
  }

1.  

    

   CloudWatch Logs Groups are detected automatically, every 10 minutes.

   Newly discovered groups start automatically to pull data. In order to pause retrieval from CWL Group, click "Edit", and "STOP" on the relevant endpoint.