If you're having trouble at any stage please contact us at support@skyformation.com.

Preface

The below guide details the major change done to the SkyFormation AWS Connector (https://support.skyformation.com/hc/en-us/articles/115001263453-SkyFormation-for-AWS-Cloud-Connector-Overview), its implications and some recommended actions to take post upgrade.

What’s changing?

Until now each AWS Connectors support one and only one specific AWS region for one specific AWS account. The region supported per connector was specified as part of the connector settings. 

Customers with AWS account that uses resources across multiple AWS regions needed to define multiple SkyFormation AWS connectors, one per region. In addition, during the connector on-boarding one had to specify a specific SQS url that the connector would monitor. This approach has two disadvantages:

1) Additional administrative work and settings to maintain and configure in SkyFormation
2) Resources added in new regions after the initial connectors setup will not be automatically monitored
3) For SQS monitoring, there was a limitation of 1 queue per connector

In order to address the above two disadvantages and allow automated way to discover and monitor AWS account with new/multiple AWS regions we have re-factored the AWS connector.

Now you will be able to configure a single AWS account, specify no specific region to monitor and the connector will automatically* discover all AWS resources which are collectable by the SkyFormation AWS Connector, in any region in the AWS account (and create endpoints for each resource).
There is no more need to create a separates AWS Connector per AWS account region you want to collect events/alerts from.

In addition, the connector would be able to auto-discover all collectable resources, including all collectable SQS URLs, not only a single one.

* In some cases as detailed below the discovery mechanism will require additional permissions

Why do you care?

This AWS Connector major change introduces few usability and technical issues that would impact any existing customer with AWS Connector. Even if you wish to keep the AWS Connector monitor a single AWS region in the AWS account as it is now. The below sections clarify the impact and actions needed. 

Is that backwards compatible?

Yes. Internally, we keep the existing monitored AWS account's region and create the existing endpoints it monitored. So all of the endpoints that the connector used to be pull data from will be created and continue to do so after the upgrade. It means the AWS account events you used to collect from the AWS Connector will be collected as before. 

Two important upgrade implications and require action to note

1. Data duplication for up to 24 hours in existing endpoints (no action required)
   As mentioned above the upgraded AWS connector will create new endpoints for all the discovered resources. The fact that the endpoints are new means that even if they collect data from the exact same resource as before, they lose their "bookmark" of the point in time where they left of before the upgrade, and start fresh, i.e. start from 24 hours before the upgrade. 
   
   For example,  if pre-upgrade we had an endpoint called “CloudTrail”, which was collecting from “us-west-2” region, and collected all events from that trail until 2 hours before the upgrade, we will now have an endpoint called “CloudTrail [us-east-2]” which is collecting the same data, but since it starts from 24 hours ago, there will be event duplication of 22 hours (from 24 hours before the upgrade and until 2 hours before the upgrade).
   
   
2. AWS SQS endpoints will stop (require manual start action by admin)
   Endpoints which collect data from AWS SQS queues whether upgraded or new are programmed to be in "inactive" state by default. The reason for this default behaviour is that this type of endpoints read their messages from the queue and delete them afterwards (this is same as it was before). So we want the AWS connector admin to do a conscious decision before starting an endpoint that removes events after collection. 
   
   Since the upgraded AWS Connector as explained above creates new endpoint for the existing upgraded AWS SQS endpoint the endpoint state will be "inactive" post upgrade.
   
   Action required: Reactivate your upgraded AWS SQS based endpoint
   Login to the SkyFormation app, go to SETTINGS-ACCOUNTS UI, allocate the SQS endpoint that will now be in "inactive" state and start it.

What else is expected?

To be multi-region and support full auto-discovery, the new connector requires a few additional AWS permissions that were not required before. (There are also a few that are not required any more).

We have added new stub endpoints called “discover status” endpoints for each AWS service, which will show the status of the auto-discovery. For a customer who did not yet provide the new permissions, those discovery endpoints will show as “ERROR” (Again, this does NOT affect the data endpoints which will continue to sync like before).

However since there will be endpoints in ERROR state, it means that the entire connector will be in ERROR state, until the newly required permissions will be provided. This is expected and does not mean there is any problem with the data itself. Once the new permissions will be given, the ERRORS will go away. The “discovery status” endpoints can also be stopped until the permissions are provided, so the connector will not show an ERROR state any more.

Once the permissions are granted, one needs to stop the connector, wait 2 minutes then start it again for the new permissions to apply.

Is there any documentation for this?

We have re-written most of the AWS documentation. Start with https://support.skyformation.com/hc/en-us/articles/360010280899

Adding AWS Cloud Connector - Multi Region version (Release 2.4.185 and later) If you're having trouble at any stage please contact us at support@skyformation.com.  Preface The goal of this guide is to add a new Amazon Web Service (AWS) connector to your SkyFormation Platform... support.skyformation.com

That document explains in details:

1. Which AWS services are collectable by skyformation
2. What configuration should take place in AWS for skyformation to pull the data
3. Which permissions are required and why
4. How to onboard the connector
   

Are the SkyFormation custom connector and AWS multi-tenant connector affected by this change?

No