Before you begin

Execute the following steps before beginning the Clustered AtScale upgrade procedure:

1. Make sure the version of AtScale you are running is 2021.3.x!

   In case you are using an earlier version you must first upgrade to 2021.3.x. For details, see Upgrading Clustered AtScale (referred to as Upgrade Guide).

2. Make sure that your AtScale Engine nodes have enough disk space.

   Note: Do not execute this check on the AtScale Coordinator node.

   As part of the upgrade procedure provided in this document, AtScale will upgrade and replicate the internal AtScale database. This can have significant disk space requirements depending on the size of the AtScale database. To check the amount of needed disk space, execute the following on each AtScale Engine node:

   1. Check the size of the Postgres database using the following command: du -sh /opt/atscale/data/database

   2. Check the available space in the Operating System partition where AtScale is installed using the following command: df -h

   3. The available space should be a minimum of 2 to 3 times greater than the current AtScale Postgres database size (as recorded in the first step).

   4. If there is not enough available space: stop, and do not attempt to move forward with the upgrade until resolving the disk space issue.

      If needed, space can be freed by removing extra log files in the /opt/atscale/log/database directory. If you are unable to free up enough space, contact AtScale Support for additional assistance to see if there are other options to proceed forward with the upgrade.

3. Perform a backup of the AtScale Database; if desired, you can also make a full backup of the AtScale system.

   It is recommended that this backup be transferred off of the host to a secure file location. For details, see Backing up and Restoring AtScale Database Instances and Backing up and Restoring AtScale.

4. Obtain certificates.

   Check if you are using Secure LDAP on AtScale, and if any certificates need to be imported into the truststore.jks or cacerts files on AtScale. If yes, obtain and store these certificates. You will need to re-import them into AtScale at the conclusion of this procedure.

5. Follow and complete the Prerequisite Installation Steps.

Procedure

1. On one of the AtScale Engine nodes, use the database/postgres_nodes tool to identify the Database Leader:

   /opt/atscale/current/bin/database/postgres_nodes

2. Based on the result from the previous step, record which host has the Leader Role.

   For the remainder of this procedure, this host will be referred to as the Database Leader node. The other nodes will be:

   • non-Leader node - this is the AScale Engine node that is not the atscale_postgres_cluster Leader, and has the role of Replica as reported by the database/postgres_nodes command.
   • Coordinator node - this is the AtScale Coordinator node.

3. Follow the steps from the Install the AtScale Installer Package section of the Upgrade Guide.

   After installing the packages, do not proceed with running the configurator.sh script as indicated on the console, and continue to follow the instructions of this procedure.

4. Using the /opt/atscale/bin/atscale_stop command, stop AtScale on all nodes in the cluster in the order in which they are listed below:

   Attention: This order must be followed!

   1. Execute an atscale_stop on the non-Leader node
   2. Execute an atscale_stop on the current Database Leader node
   3. Execute an atscale_stop on the Coordinator node

5. Only on the Database Leader node, obtain the system locale by running the localectl command.

   For example, you can see the following result: System Locale: LANG=en_US.UTF-8

   The locale is identified as the value after LANG=. In the above example, the system locale is en_US.UTF-8.

6. Only on the Database Leader node, run the installer/database-upgrade.sh tool from the newly installed (but not yet activated) AtScale version directory.

   This tool will port data from the current PostgreSQL 9.6 directory the new PostgreSQL 14.0 directory.

   If during the previous step the localectl command returned a value, then the --locale argument is needed and the tool should be run as follows:

   /opt/atscale/versions/<NEW_2021.4.X_ATSCALE_VERSION>/bin/installer/database-upgrade.sh --locale "<locale value>"

   Here is an example (with locale en_US.UTF-8):

   /opt/atscale/versions/2021.4.0.4227/bin/installer/database-upgrade.sh --locale "en_US.UTF-8"

   When asked to continue enter y.

   Attention: The tool must only be run on the Database Leader node. The Database Standby nodes will be migrated automatically to the new data directory /opt/atscale/data/database-14 upon AtScale upgrade. If the tool was accidentally run on a Database Standby node, simply delete the /opt/atscale/data/database-14 directory before activating AtScale on that node.

7. Verify that the database-upgrade.sh script completed successfully and there were no errors or failures reported; the logs are at /opt/atscale/log/database/database_upgrade.

   Example for successful completion:

   Upgrade Complete

   ----------------

   Optimizer statistics are not transferred by pg_upgrade.

   Once you start the new server, consider running:

   /var/atscale/versions/2021.4.0.4227/pkg/postgresql/bin/vacuumdb -U atscale_super --all --analyze-in-stages

   
   

   Running this script will delete the old cluster's data files:

   ./delete_old_cluster.sh

   
   

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Cleaning out /var/atscale/data/coordinator

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   
   

   /var/atscale/data/coordinator cleaned

   
   

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Cleaning out /var/atscale/data/service_registry

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   
   

   /var/atscale/data/service_registry cleaned

   Example for unsuccessful completion:

   ERROR *** pg_upgrade --check failed with exit code: 1. Has the database already been upgraded?

   Please see logs in /var/atscale/log/database/database_upgrade for more info.

   In case the tool failed: review the failure message, and then check the More information section below for next steps.

8. On the non-Leader node and on the Coordinator node, delete the following directories:

   rm -rf /opt/atscale/data/coordinator

   rm -rf /opt/atscale/data/service_registry/

   Note: Do not delete these directories on the Database Leader node!

After completing the database upgrade you need to activate AtScale as described in the next section.

Activating AtScale

1. Activate AtScale on the Coordinator node.

   1. As the atscale user, run the installed configurator.sh script in --activate mode:

      su - $ATSCALE_USER

      cd /opt/atscale/versions/<package_version>

      ./bin/configurator.sh --activate

   2. Enter y when asked "Do you want to apply these settings?"

   3. If prompted with the following message, enter N to proceed with the activation. This is normal, since on the AtScale Coordinator node there is no database-14 file or directory.

      The old AtScale database directory, /opt/atscale/data/database,

      exists and the new one, /opt/atscale/data/database-14,

      has either not yet been created or is still empty.

      
      

      If you are upgrading AtScale, please exit and run

      /opt/atscale/versions/2021.4.0.4227/bin/installer/database-upgrade.sh

      before continuing with the Configurator.

      
      

      Do you wish to exit in order to run /opt/atscale/versions/2021.4.0.4227/bin/installer/database-upgrade? (y/N): N

   4. Once the activation is completed proceed to the next step.

2. Activate AtScale on the Database Leader node.

   1. As the atscale user, run the installed configurator.sh script in --activate mode:

      su - $ATSCALE_USER

      cd /opt/atscale/versions/<package_version>

      ./bin/configurator.sh --activate

   2. Enter y when asked "Do you want to apply these settings?"

   3. Once the activation is completed proceed to the next step.

3. Activate AtScale on the non-Leader node.

   1. As the atscale user, run the installed configurator.sh script in --activate mode:

      su - $ATSCALE_USER

      cd /opt/atscale/versions/<package_version>

      ./bin/configurator.sh --activate

   2. Enter y when asked "Do you want to apply these settings?"

   3. When prompted with the following message, enter N to proceed with the activation. This is normal and expected, since on the AtScale non-Leader node there is no database-14 file or directory.

      The old AtScale database directory, /opt/atscale/data/database,

      exists and the new one, /opt/atscale/data/database-14,

      has either not yet been created or is still empty.

      
      

      If you are upgrading AtScale, please exit and run

      /opt/atscale/versions/2021.4.0.4227/bin/installer/database-upgrade.sh

      before continuing with the Configurator.

      
      

      Do you wish to exit in order to run /opt/atscale/versions/2021.4.0.4227/bin/installer/database-upgrade? (y/N): N

4. Import certificates.

   If Secure LDAP is used in your AtScale environment, and if the LDAP certificates were imported into AtScale to allow connection to the Secure LDAP environment, then the certificates will need to be re-imported into both the Database Leader and non-Leader Engine nodes. Follow this procedure:

   1. Execute the atscale_stop command on the non-Leader node.
   2. Execute the atscale_stop command on the Database Leader node.
   3. On the Database Leader node, import the certificates into AtScale's cacerts file and run the configurator.sh -apply command as described in Before you begin: Importing a Certificate.
   4. Repeat the previous step on the non-Leader node.
   5. Verify that LDAPS connectivity is working.

5. Complete the upgrade of AtScale by following the steps from the What to do next section of the Upgrade Guide.

More information

Note the following about the database-upgrade.sh tool:

• Running it more than once is fine, since subsequent runs will fail with a warning stating that the database has already been initialized and is likely upgraded.

• If a full re-run of the tool is needed, first move or delete the /opt/atscale/data/database-14 and then run the tool again.

• If the tool continues to fail, contact AtScale Support for assistance.

• In case the tool fails and you want to restore AtScale, the AtScale services can be restarted using the /opt/ascale/bin/atscale_start command in the standard order:

  1. AtScale Coordinator node
  2. AtScale Database Leader node
  3. AtScale non-Leader node

• If you want to see additional details while running the tool you can use the --help argument.

  /opt/atscale/versions/<NEW_2021.4.X_ATSCALE_VERSION>/bin/installer/database-upgrade.sh --help

Consider that the old /opt/atscale/data/database directory will no longer be used after the AtScale upgrade. It can be removed once everything in the upgraded AtScale is running as expected.