How-to: Configure SkyFormation and AWS to Monitor AWS S3/Lambda Data Events

Data plane events, as opposed to management events, are not monitored by CloudTrail out-of-the box. But, CloudTrail can be configured so to monitor these events.

We will use this capability in order to direct these events into an SQS queue, from which SkyFormation can pull them at near real-time latency.

Configure CloudTrail and CloudWatch Events to monitor data events

In order to enable this monitoring capability -

1. Go to a CloudTrail's trail configuration
   

2. Under "Data events", click edit, under S3 tab, check the boxes for the S3 buckets you'd like to monitor (or check checkbox for All S3 buckets to monitor all exisiting and future buckets). Select which actions to monitor (Read, Write).
   Repeat the process for the Lambda functions under the "Lambda" tab
   
   

3. Click "Save"

4. Create an SQS queue that will be the target of the above events. We will configure it as a target in the next steps.

5. Go to CloudWatch Events, and create a new rule -
   1. Events Source is "Event Pattern", for service "Simple Storage Service (S3)" and another rule for "Lambda", Event Type is "All Events"
   2. Targets - add a target of "SQS queue", and select the queue created in step #4
   3. Continue by clicking "Configure Details"
   4. Give it a meaningful name, i.e. "CloudTrail Data Events", keep the checkbox of state enabled, and click "Create rule"
      
      

6. Verify the configuration is working by forcing some event to occur (i.e. downloading an object from a monitored S3 bucket), and verify it is present in the queue

Configure SkyFormation's AWS account to monitor the pipeline

1. Configure SkyFormation to monitor this queue by adding its URL to the AWS Account credentials -
   1. Login to SkyFormation instance
   2. Go to Settings -> Accounts -> Edit the AWS Account
   3. Under Sqs-Url set the URL of the SQS queue created at step #4 (and used in step #5.2)
   4. Click Save
   5. Click "Status" on the AWS Account, and verify that the endpoint "cloud-watch-events-sqs" is started (or click "Start" to start it up)

Troubleshoot

• Verify that the credentials given to the user/access key that were provided to SkyFormation has permission to read and write on the above SQS queue.