How-To: Handle Out of Disk Space (versions 2.4.x)

The goal of the following procedure is, in the case out of disk space due to large number of events temporarily persisted in Kafka (retention period is by default 24HR), where you can't allocate more space - to clear current Kafka data while making sure all events where pushed to the syslog receiver and decrease Kafka's data retention period to prevent it from recurring. 

• SK4_ROOT_DIR - is /opt/exabeam/data/sk4 for Exabeam default installation and /opt/sk4 for non Exabeam default installation 
• Location of a volume can be found by running

  docker volume inspect <VOLUME_NAME>

Procedure to Decrease Kafka's Data Retention Period

1. Deactivate all accounts via the SK4 Web UI
2. Wait 2min
3. Stop the SkyFormation server

   systemctl stop sk4compose

4. Delete content of the sk4_kafka_data volume
5. Delete Kafka's offset tracking information in ZooKeeper 
   

   docker-compose -f <SK4_ROOT_DIR>/docker-compose.yml up -d zookeeper

   docker container exec -it sk4zookeeper zkCli.sh

   deleteall /sk4kafka

   quit

   docker-compose -f <SK4_ROOT_DIR>/docker-compose.yml down

6. Delete server.properties in the sk4_kafka_conf volume
7. Change Kafka's data retention period by updating the KAFKA_LOG_RETENTION_MS environment variable at the <SK4_ROOT_DIR>/docker-compose.yml file (.services.kafka.environment[KAFKA_LOG_RETENTION_MS])
8. Start the SkyFormation server

   systemctl start sk4compose