How to reduce the size of events sent to your SIEM

Preface

In this guide we’ll cover the structure of events sent to your SIEM, when in CEF format, and learn how to remove parts of it that are unnecessary to your needs, such that the total event size of each event is reduced.

Event structure

The cloud connectors module (i.e. SkyFormation) can send events to your SIEM in 2 formats - CEF or JSON. When data is sent as CEF, each event is composed of 3 main parts. 1. Data that is extracted by the application from the event is set on select CEF fields, per event type. Not all data items are extracted, only ones that have security value and are used later for analysis. They are set on CEF field with disregard to the original meaning of the CEF field, but as a placeholder. 2. The raw event, as it was received by the application from the Cloud vendor. In most cases is it a JSON object, but sometimes it is not. It is put on the CEF field “cs6”, i.e. Custom String 6. 3. The raw event flattened as key=value pairs. Often SIEM system have a hard time extracting data from JSON structures. To ease that pain, we’ve flattened the raw event’s data for easier consumption for such systems. This only applies for raw events that are in JSON structure. An example raw event might look like this:

{
    "top": "top-value",
    "h0": {
        "h1": "deep-value"
    }
}

would yield the following CEF fields. the _ext prefix added for collision prevention:
_ext_top=top-value
_ext_h0_h1=deep-value

Removing unnecessary parts

As raw events sent from certain cloud vendors can reach tens of kilobytes, you might want to reduce the event size sent to your SIEM such that either the raw event itself, or the flattening of it, or both, not be sent to your SIEM.

Beware that by removing the raw event, you will later not be able to investigate/query for it in your SIEM. The events are not stored in the Cloud Connectors module for later inquiry.

In order to remove any of the above event parts:

1. Login to your Cloud Connectors instance, navigate to Settings -> Advanced (top tight tab)

2. In the input field put:

For CC version 2.4.x:

/sk4/tenants/default-tenant-id/settings/cef-transformation-settings

For CC version 2.5.x:

/sk4/tenants/default-tenant-id/settings/CefTransformationSettings/cef-transformation-settings

NOTE: due to a regression in 2.5.0-2.5.129 this setting was ignored. Make sure CC version is 2.5.130 or above is in use, or upgrade

3. Click on the blue triangle to the right of the input field.

4. Insert the below JSON, and click “SAVE” at the bottom. Once saved, restart the application to take effect – Via the terminal on the machine where Cloud Connectors is installed run
sudo systemctl restart sk4compose

{
    "send-raw-event": true,
    "flatten-raw-event": false
}

Modify this JSON to your needs

Set "send-raw-event" to false to NOT include the raw event from the cloud vendor in the CEF. the cs6 field will not be set with any value

Set "flatten-raw-event" to false to NOT include the flattened _ext* fields of the raw event.

These two settings are independent, and you can choose to set either to meet your needs.