Replacing a Clustered AtScale Machine

This document describes how to replace a single machine in an AtScale cluster.

Before you begin

These instructions are for CentOS/RedHat operating systems. You may have to adapt some commands to your supported operating system.
Unless otherwise specified, run commands as atscale user.

About this task

Machines required

Running Clustered AtScale requires a customer-provided external load balancer and three AtScale machines: one that will run AtScale Coordinator (a stripped-down version of AtScale), and two that will run the full AtScale application both in clustered mode. In addition, these instructions refer to a Replacement Host that will replace an existing cluster host.

These AtScale machines will be referred to as follows:

Application Host 1 - runs full AtScale
Application Host 2 - runs full AtScale
Coordinator Host - runs the AtScale Coordinator
Replacement Host - new Host that will replace one of the other hosts

Replacing an AtScale Host with a Host that has an Identical DNS Fully Qualified Domain Name (FQDN)

Replacing a single clustered AtScale machine using the identical DNS FQDN is a simpler process than replacing an AtScale cluster host with a different FQDN. With in the context of the CONVERT STAND-ALONE ATSCALE TO CLUSTERED ATSCALE Instructions, the replacement host should be thought of as a "standby" host since a "master" is already running. Follow these steps:

If the host you want to replace is still running then shut it down.
Copy the installation package to the replacement host. As root Install the AtScale installer package, but do not run the confgurator.sh script. If you are not root, but have sudo privileges install the rpm with the sudo -E option to preserve the ATSCALE_USER environment variable in the subshell.
rpm -i installer_package_filename.rpm
or if using Debian packages:
dpkg --install installer_package_filename.deb
Copy the /opt/atscale/conf/atscale.yaml from the running application host to /opt/atscale/conf/atscale.yaml on the replacement host.
Activate the replacement host:
1. On the replacement host, as the ATSCALE_USER user (atscale by default), run the installed AtScale's configurator.sh script in --activate mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g. su - myatscaleuser):
  su - atscale
  cd /opt/atscale/versions/<package_version>
  ./bin/configurator.sh --activate
2. You will be asked to confirm that you want to apply this configuration. Answer y to start the activation process, which will configure and run AtScale.
  Note: If an error occurs, you must perform a full reinstallation. You cannot simply rerun the installer. See Uninstall AtScale.

Replacing an AtScale Host with a Host that has a Different DNS Fully Qualified Domain Name (FQDN)

Replacing an AtScale host using a different DNS FQDN requires that AtScale be installed and activated on the new machine and reconfigured on the two machines still running AtScale.

1. On the new machine, install, but do not activate, the AtScale installer package.

Follow the Stand-Alone installation instructions, but do not run the Activate AtScale step.

2. Identify the AtScale Application Host that's the current Database Leader.

Identify the current Database Leader member by running the patronictl utility on one of the AtScale Application Hosts.

$ /opt/atscale/current/bin/database/patronictl list
+--------------------------+------------+------------+--------+---------+-----------+
| Cluster | Member | Host | Role | State | Lag in MB |
+--------------------------+------------+------------+--------+---------+-----------+
| atscale_postgres_cluster | atscale-01 | atscale-01 | Leader | running | |
| atscale_postgres_cluster | atscale-02 | atscale-02 | | running | |
+--------------------------+------------+------------+--------+---------+-----------+

3. Stop AtScale on each of the machines where it's still running.

Stop AtScale on each of the machines where it's still running, in the following order. If AtScale is currently not running on a given machine, you can proceed to the next one in the list.

AtScale Application Host that is not the current Database Leader
AtScale Application Host that is the current Database Leader
AtScale Coordinator

Stop AtScale:

$ /opt/atscale/bin/atscale_stop

4. Edit /opt/atscale/conf/atscale.yaml.

On the AtScale Application Host that is the current Database Leader, edit /opt/atscale/conf/atscale.yaml, changing the information for the host that's being replaced to that of the host that's being newly introduced.

Copy the modified /opt/atscale/conf/atscale.yaml from the AtScale Application Host that is the current Database Leader to /opt/atscale/conf/atscale.yaml on both of the other hosts.

5. Activate or reconfigure AtScale Coordinator.

If the AtScale Coordinator host is the one being replaced, it will need to be activated. Otherwise it can just have the updated configuration applied.

Activate the "Coordinator" host
1. On the "Coordinator" host, as the ATSCALE_USER user (atscale by default), run the installed AtScale's configurator.sh script in --activate mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g. su - myatscaleuser):
  su - atscale
  cd /opt/atscale/versions/<package_version>
  ./bin/configurator.sh --activate
2. You will be asked to confirm:
1. That you want to apply this configuration
  1. Answer y to start the activation process, which will configure and run AtScale.
    Note: If an error occurs, you must perform a full reinstallation. You cannot simply rerun the installer. See Uninstall AtScale.
Apply config to the "Coordinator" host
1. On the "Coordinator" host, as the ATSCALE_USER user (atscale by default), run the installed AtScale's configurator.sh script in --apply mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g. su - myatscaleuser):
  su - atscale
  cd /opt/atscale/versions/<package_version>
  ./bin/configurator.sh --apply
2. You will be asked to confirm:
  1. That you want to apply this configuration
3. Answer y to start the config apply process, which will configure and run AtScale.

6. Reconfigure the AtScale Application Host that's the Database Leader.

The Database Leader host will never be the one that's being replaced and will therefore only need to be reconfigured.

Apply config to the "Database Leader" host
1. On the "Database Leader" host, as the ATSCALE_USER user (atscale by default), run the installed AtScale's configurator.sh script in --apply mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g. su - myatscaleuser):
  su - atscale
  cd /opt/atscale/versions/<package_version>
  ./bin/configurator.sh --apply
2. You will be asked to confirm:
  1. That you want to apply this configuration
3. Answer y to start the config apply process, which will configure and run AtScale.

7. Activate or reconfigure remaining AtScale Application Host.

If the remaining AtScale Application host is the one being replaced, it will need to be activated. Otherwise it can just have the updated configuration applied.

Activate the remaining AtScale Application host
1. On the remaining AtScale Application host, as the ATSCALE_USER user (atscale by default), run the installed AtScale's configurator.sh script in --activate mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g. su - myatscaleuser):
  su - atscale
  cd /opt/atscale/versions/<package_version>
  ./bin/configurator.sh --activate
2. You will be asked to confirm:
1. That you want to apply this configuration
  1. Answer y to start the activation process, which will configure and run AtScale.
    Note: If an error occurs, you must perform a full reinstallation. You cannot simply rerun the installer. See Uninstall AtScale.
Apply config to the remaining AtScale Application host
1. On the remaining AtScale Application host, as the ATSCALE_USER user (atscale by default), run the installed AtScale's configurator.sh script in --apply mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g. su - myatscaleuser):
  su - atscale
  cd /opt/atscale/versions/<package_version>
  ./bin/configurator.sh --apply
2. You will be asked to confirm:
  1. That you want to apply this configuration
3. Answer y to start the config apply process, which will configure and run AtScale.

8. Update the External Load Balancer Configuration.

If replacing an application host, update your external load balancer configuration to reference the replacement host name rather than the host that was removed from the cluster. This step is not necessary if you are replacing a Coordinator host.

What to do next

You may now use the AtScale cluster.