How to migrate an existing deployment to another server

Background & Prerequisites

When migrating from a source system to a target system, both systems must be on the same SW version of Skyformation Cloud Connectors.

Versions 2.4.x of Skyformation use zookeeper for configuration management while versions 2.5.x moved to using etcd. It is highly recommended that you upgrade your existing systems to the latest Skyformation version (2.5.x) using etcd before proceeding with the migration. If you need to remain on your current version, you may do so, however you will need to contact support to get the installation command to your specific version.

If you wish to upgrade your source system before continuing to do the migration please follow the instructions here --> https://support.skyformation.com/hc/en-us/articles/115004121433-Updating-SkyFormation-app-on-a-Linux-system

The next section will provide instructions on how to perform this operation in an automatic way. If you prefer to perform it manually, you may follow the guide of the next section. In both of the sections there are steps to migrate configurations from one Skyformation instance (the source) to another (the target). As mentioned in the prerequisites section, the migration must be done between instances with exactly the same code version.

Automatic Migration of Skyformation from a source system to a target system

1. In the source system, please run the following command:

 sudo bash -c "$(curl -L https://download.skyformation.net/v2/download/installer/edge/migration_export.sh)" $@ 2>&1 | tee "migration-export-at-`date --iso-8601=ns`.log"

You will be asked to enter a path where your configurations will be saved. Please make sure it is a non-existing directory. 

You should see the message: "==> Done export! The backup dir is in:<Your backup directory>" when the process is finished.

2. Copy the directory created in step 1, from the source system to the target system (using scp or  rsync for example)

3. In the target system, please run the following command:

sudo bash -c "$(curl -L https://download.skyformation.net/v2/download/installer/edge/migration_import.sh)" $@ 2>&1 | tee "migration-import-at-`date --iso-8601=ns`.log"

You will be asked to enter the path to the directory that was copied in step 2.

You should see the messages "Services are running successfully" and then "==> Done import!"

when the process is finished. 

Manual Migration of Skyformation from a source system to a target system

Please apply instructions that matches your migration path.

1. On the source system, go to the Cloud Connectors UI , settings --> accounts and stop all the active accounts one by one, then wait at least 5 minutes to let it gracefully stop. You may want to take a note of the accounts that were active.

2. Remove any old utilities image from the system: on the source system terminal, type: 

   • sudo docker image rm -f skyformation/troubleshoot-utils 

   • If the image is not present you will get the following error : 
     Error: No such image: skyformation/troubleshoot-utils 
     this is OK.
3. Export source settings:
   On the source host, run the following based on your migration path:
   
   1. If your systems (source&target) run Skyformation 2.5.x, apply the following command to send configurations into a sk4etcdkbackup- folder (the folder will be created in the same directory where you ran the command):

      1.  sudo docker run --rm --network=sk4_isolated_nw -v "$PWD/sk4etcdbackup-$(date +%Y-%m-%d-%H-%M-%S)":/local skyformation/troubleshoot-utils:latest etcd-export /local 

   2. If you are migrating to continue using Skyformation 2.4.x, run the following command to send configurations into a sk4zookbackup-<date> folder(the folder will be created in the same directory where you ran the command):

      1.  sudo docker run --rm --network=sk4_isolated_nw -v "$PWD/sk4zookbackup-$(date +%Y-%m-%d-%H-%M-%S)":/local skyformation/troubleshoot-utils:latest zookeeper-export /local

4. Version 2.5.x only : Install Skyformation on the target host with the same Skyformation version as the source. To confirm the source version, in the Skyformation UI go to settings --> general. To install skyformation with a specific SW version follow the instructions here:  https://support.skyformation.com/hc/en-us/articles/360014408940-Install-a-specific-Skyformation-version-on-Linux-Machine
   If you need to install version 2.4 on your target please contact support to get the command to install old version.
5. Stop the SK4 service on both the source and the target:

   1. sudo systemctl stop sk4compose

6. Override the encryption key in the target with the one from the source
   1. Find the location of the docker-compose.yml file on the source and target systems and if an override file exists (source only)
      1. On the terminal of each system, type: 

      2. cat /etc/systemd/system/sk4compose.service | grep ExecStart 

      3. In the output line, the compose file path shows after the -f. If more than 1 file appears as a parameter -f, the first one is the compose file and the second one is the override file.
         An example result:
         

         ExecStart=/usr/local/bin/docker-compose -f /opt/sk4/docker-compose.yml up
         
         The docker-compose.yml file full path is /opt/sk4/docker-compose.yml

      4. NOTE: the location of the docker-compose.yml file may be different between the source and the target. This is OK. You should not change the location of the file on any system.
         
         
   2. Find the SKYFORMATION_ENC_KEY string from the source and target machines. Type:

      1. grep SKYFORMATION_ENC_KEY <compose file full path> | uniq | cut -f2- -d=

   3. Replace the SKYFORMATION_ENC_KEY env variable in the target machine (using the key from the source machine). On the target machine type:

      1. sudo sed -i 's/<CURRENT_KEY_IN_TRG>/<OLD_KEY_FROM_SOURCE>/g' <target system compose file full path>

      2. Repeat the command from 2.1.1 to get the SKYFORMATION_ENC_KEY on the target to make sure it was successfully replaced.
   4. If an override file exists on the source machine, copy it to the directory where the compose resides on the target system and edit the file /etc/systemd/system/sk4compose.service to add the additional -f flag with the override file, similar to what was in the source system.
      
      
7. In this step we are going to copy the contents of 3 docker volumes from the source system to the target system: sk4_conf, sk4_pg_data, sk4_pg_conf
   1. First, find out the location of these docker volumes on the source and target systems – it is not necessarily in the same path! modify only the content, do not change the path/location of the volume on the target system.
      To find the path, type (source and target systems):

      1. sudo docker volume inspect sk4_conf

      2. The path is listed under the “Mountpoint” entry. The 3 volumes we need to copy are all located under the same root path.
   2. Repeat the next step for each of the 3 volumes:
      1. Copy the *contents* of the directory from the source system into their corresponding locations on the target system (using scp or rsync for example)
   3. Check file permissions on YML and SH files in the sk4_conf directory on the target:
      1. For the SH file, the execute flag should be enabled. If not, run chmod +x *.sh to configure file permissions.
      2. For the YML files, read+write access is required for the user account that runs Skyformation.
         
         
8. On the target system, start the configuration service based on your installation version
   1. If the target is running Skyformation 2.5.x (or later) run:

      1. sudo /usr/local/bin/docker-compose -f <docker compose yml full path> up -d sk4etcd 

   2. If the target is running Skyformation 2.4.x :

      1. sudo /usr/local/bin/docker-compose -f <docker compose yml full path>  up -d zookeeper 

9. Import configurations into the target, based on your installation:
   1. Copy the sk4etcdkbackup-<date> or sk4zookbackup-<date> folder generated in Step 3.1 from the source system to the local home folder ($PWD) on the target.
   2. Run the command for the your migration path:
      1. If the target is on Skyformation 2.5.x, import the etcd contents using the sk4etcdkbackup-<date> folder:

      2. sudo docker run --rm --network=sk4_isolated_nw -v "$PWD/sk4etcdbackup-<date that was created during the import>":/local skyformation/troubleshoot-utils:latest etcd-import /local 

      3. Run the following command to ensure data has been imported successfully. You should see entries (more than one) for the imported data, specifically there should be an entry for each account that was imported:

         1.  sudo docker container exec -it sk4etcd etcdctl get "" --prefix=true --keys-only

      4. If the target is on Skyformation 2.4.x, import the zookeeper contents using the sk4etcdkbackup- folder:

         1. sudo docker run --rm --network=sk4_isolated_nw -v "$PWD/sk4zookbackup-<date> ":/local skyformation/troubleshoot-utils:latest zookeeper-import /local

10. Start Skyformation on the target:

    1. sudo systemctl start sk4compose

11. On the Skyformation Cloud Connectors UI on the target, start all the accounts that were active on the source. 
    
    
12. After verifying data is flowing on the target, stop and disable Skyformation on the source:

13. sudo systemctl stop sk4compose
    sudo systemctl disable sk4compose