See: Adding SkyFormation for AWS Cloud Connector guide 

 

(b) Install and configure AWS CLI 
     See https://aws.amazon.com/cli/

(c) Associate a subscription filter with Kinesis to the CloudWatch Logs Group used for
     the VPC Flow Logs

     To create a subscription filter for Kinesis

> NOTE: Create the Kinesis stream in the same region as the CloudWatch Logs are for faster and easier configuration. Replace the region in the example below with the real region name, i.e. us-east-1

1. Create a destination Kinesis stream (e.g. named VpcFlowLogsStream) using the following command: 

   $ C:\>  aws kinesis create-stream --stream-name "VpcFlowLogsStream" --shard-count 1

2. Wait until the Kinesis stream becomes Active (this might take a minute or two). You can use the following Kinesis describe-stream command to check the StreamDescription.StreamStatus property.
   
   In addition, note the StreamDescription.StreamARN value, as you will need it in a later step: 

   aws kinesis describe-stream --stream-name "VpcFlowLogsStream"

   The following is example output: 

   {
       "StreamDescription": {
           "StreamStatus": "ACTIVE",
           "StreamName": "VpcFlowLogsStream",
           "StreamARN": "arn:aws:kinesis:region:123456789012:stream/VpcFlowLogsStream",
           "Shards": [
               {
                   "ShardId": "shardId-000000000000",
                   "HashKeyRange": {
                       "EndingHashKey": "340282366920938463463374607431768211455",
                       "StartingHashKey": "0"
                   },
                   "SequenceNumberRange": {
                       "StartingSequenceNumber":
                       "49551135218688818456679503831981458784591352702181572610"
                   }
               }
           ]
       }
   }

3. Create the IAM role that will grant CloudWatch Logs permission to put data into your Kinesis stream. First, you'll need to create a trust policy in a file
   (for example, ~/TrustPolicyForCWL.json).
   
   Use a text editor to create this policy. Do not use the IAM console to create it. 

   {
     "Statement": {
       "Effect": "Allow",
       "Principal": { "Service": "logs.region.amazonaws.com" }, 
    "Action": "sts:AssumeRole" 
    } 
   }

4. Use the create-role command to create the IAM role, specifying the trust policy file. Note the returned Role.Arn value, as you will also need it for a later step:

   aws iam create-role --role-name CWLtoKinesisRole --assume-role-policy-document file://~/TrustPolicyForCWL.json

   {
    "Role": {
    "AssumeRolePolicyDocument": {
    "Statement": {
    "Action": "sts:AssumeRole",
    "Effect": "Allow",
    "Principal": {
    "Service": "logs.region.amazonaws.com"
    }
    }
    },
    "RoleId": "AAOIIAH450GAB4HC5F431",
    "CreateDate": "2015-05-29T13:46:29.431Z",
    "RoleName": "CWLtoKinesisRole",
    "Path": "/",
    "Arn": "arn:aws:iam::123456789012:role/CWLtoKinesisRole"
    }
   }

5. Create a permissions policy to define what actions CloudWatch Logs can do on your account. First, you'll create a permissions policy in a file (for example, ~/PermissionsForCWL.json). Use a text editor to create this policy. Do not use the IAM console to create it. 

   {
    "Statement": [
    {
    "Effect": "Allow",
    "Action": "kinesis:PutRecord",
    "Resource": "arn:aws:kinesis:region:123456789012:stream/VpcFlowLogsStream"
    },
    {
    "Effect": "Allow",
    "Action": "iam:PassRole",
    "Resource": "arn:aws:iam::123456789012:role/CWLtoKinesisRole"
    }
    ]
   }

6. Associate the permissions policy with the role using the following put-role-policy command: 

   aws iam put-role-policy --role-name CWLtoKinesisRole --policy-name Permissions-Policy-For-CWL --policy-document file://~/PermissionsForCWL.json

7. After the Kinesis stream is in Active state and you have created the IAM role, you can create the CloudWatch Logs subscription filter. The subscription filter immediately starts the flow of real-time log data from the chosen log group to your Kinesis stream:

   aws logs put-subscription-filter \
       --log-group-name "VpcFlowLogsGroup" \
       --filter-name "VpcFlowLogsStream" \
       --filter-pattern "" \
       --destination-arn "arn:aws:kinesis:region:123456789012:stream/VpsFlowLogsStream" \ --role-arn "arn:aws:iam::123456789012:role/CWLtoKinesisRole"

8. After you set up the subscription filter, CloudWatch Logs forwards all the incoming log events that match the filter pattern to your Kinesis stream.

9. Verify that the VPC Flow Logs are forward to the Kinesis stream as intended.
   
   Run the Kinesis get-records command to fetch some Kinesis records:

   aws kinesis get-shard-iterator --stream-name VpcFlowLogsStream --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON

   {
       "ShardIterator":
       "AAAAAAAAAAFGU/kLvNggvndHq2UIFOw5PZc6F01s3e3afsSscRM70JSbjIefg2ub07nk1y6CDxYR1UoGHJNP4m4NFUetzfL+wev+e2P4djJg4L9wmXKvQYoE+rMUiFq+p4Cn3IgvqOb5dRA0yybNdRcdzvnC35KQANoHzzahKdRGb9v4scv+3vaq+f+OIK8zM5My8ID+g6rMo7UKWeI4+IWiK2OSh0uP"
   }

   aws kinesis get-records --limit 10 --shard-iterator "AAAAAAAAAAFGU/kLvNggvndHq2UIFOw5PZc6F01s3e3afsSscRM70JSbjIefg2ub07nk1y6CDxYR1UoGHJNP4m4NFUetzfL+wev+e2P4djJg4L9wmXKvQYoE+rMUiFq+p4Cn3IgvqOb5dRA0yybNdRcdzvnC35KQANoHzzahKdRGb9v4scv+3vaq+f+OIK8zM5My8ID+g6rMo7UKWeI4+IWiK2OSh0uP"

   Note that you might need to make this call a few times before Kinesis starts to return data.

   You should expect to see a response with an array of records. The Data attribute in an Kinesis record is Base64 encoded and compressed with the gzip format. You can examine the raw data from the command line using the following Unix commands: 

   echo -n "<Content of Data>" | base64 -d | zcat

   The Base64 decoded and decompressed data is formatted as JSON with the following the key elements:
   
   owner

   The AWS Account ID of the originating log data.

   logGroup

   The log group name of the originating log data.

   logStream

   The log stream name of the originating log data.

   subscriptionFilters

   The list of subscription filter names that matched with the originating log data.

   messageType

   Data messages will use the "DATA_MESSAGE" type. Sometimes CloudWatch Logs may emit Kinesis records with a "CONTROL_MESSAGE" type, mainly for checking if the destination is reachable.

   logEvents

   The actual log data, represented as an array of log event records. The "id" property is a unique identifier for every log event.

 

(d) Give the IAM user used for your SkyFormation for AWS connector the additional permissions
     to allow it to interact with the Kinesis stream created.

     

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "VisualEditor0",
 "Effect": "Allow",
 "Action": [
 "kinesis:GetShardIterator",
 "kinesis:GetRecords",
 "kinesis:DescribeStream"
 ],
 "Resource": "arn:aws:kinesis:region:123456789012:stream/VpcFlowLogsStream"
 }
 ]
}

Configure the SkyFormation for AWS connector 

(a) Login to your SkyFormation app

 

(b) Navigate to SETTINGS->ACCOUNTS

 

(c) Focus on the AWS connector you would like to add the new VPC Flow Logs stream using the
     Kinesis stream to and press its EDIT button and scroll to the list of the Connector's endpoints.

 

(d) Locate an endpoint that starts with
     "ENDPOINT - CLOUDWATCH-LOGS - [%the VPC Flow Logs group streamed to kinesis%]

 

     For example in the snapshot below we looked for VPC Flow Logs group named:
     "TMP-FLOW-LOGS" 

 

(e In the "Stream-Name" box inser the name of the Kinesis stream you used to stream the 
      VPC Flow Logs into (e.g. VpcFlowLogs in our example)

 

(f) Press DONE

 

(g) Focus on the AWS connector you configured, click the STATUS button,
     and scroll to the list of the Connector's endpoints.