Adding a New Data-Center Into an Existing Scylla Cluster

The following procedure specifies how to add a Data Center (DC) to a live Scylla Cluster, single or multi-data-center. Adding a DC out scale the cluster, providing higher availability. The procedure includes:

  • Installing nodes on the new DC.
  • Adding the new nodes to the cluster.
  • Update the replication strategy of selected keyspace to use the new DC.
  • Rebuild.

Prerequisites

  1. Login to one of the nodes in the cluster, collect the following info from the node:
  • cluster_name - cat /etc/scylla/scylla.yaml | grep cluster_name
  • seeds - cat /etc/scylla/scylla.yaml | grep seeds:
  • endpoint_snitch - cat /etc/scylla/scylla.yaml | grep endpoint_snitch
  • Scylla version - scylla --version

2. Install the new Scylla nodes on the new data-center, see Getting Started for further instructions, create as many nodes that you need. Follow the Scylla install procedure up to scylla.yaml configuration phase. In the case that the node starts during the process follow these instructions.

  1. Select couple of nodes to be seeds nodes

Procedure

  1. Alter each Keyspace replication to use class : NetworkTolologyStrategy and set the new DC replication factor to zero. This will prevent writing to the new DC until explicitly enabled.

Alter the following:

  • Keyspace created by the user
  • System: system_auth, system_distributed, system_traces

For example:

Before

ALTER KEYSPACE <keyspace> WITH REPLICATION =
  { 'class' : 'SimpleStrategy', '<exiting_dc>' : 3};

After

ALTER KEYSPACE <keyspace> WITH REPLICATION =
  { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc>:0};
  1. Configure each node scylla.yaml file in the existing data-center(s) to be either of:
  • Ec2MultiRegionSnitch - for AWS cloud-based, multi-data-center deployments
  • GossipingPropertyFileSnitch - for bare metal and cloud (other than AWS) deployments
  1. Update scylla.yaml file in the existing data-center(s) with the newly promoted seed nodes.

Retart Scylla node one by one using

sudo systemctl restart scylla-server

4. For each node in the new data-center edit the scylla.yaml file parameters listed below, the file can be found under /etc/scylla/

  • cluster_name - Set the selected cluster_name
  • listen_address - IP address that Scylla used to connect to the other Scylla nodes in the cluster
  • seeds - Set the selected seed nodes - add new DC seeds in both the old DC and in the new DC
  • auto_bootstrap - By default, this parameter is set to true, it allow new nodes to migrate data to themselves automatically, set the auto_bootstrap to false to prevent data streaming
  • endpoint_snitch - Set the selected snitch
  • rpc_address - Address for client connections (Thrift, CQL)
  • The parameters seeds, cluster_name and endpoint_snitch need to match the existing cluster.
  1. In the new data-center, start Scylla nodes one by one using

sudo systemctl start scylla-server

  1. Verify that the nodes were added to the cluster using nodetool status

For example:

$ nodetool status

Datacenter: US-DC
=========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--   Address         Load            Tokens  Owns            Host ID                                 Rack
UN   54.191.2.121    120.97 KB       256     ?               c84b80ea-cb60-422b-bc72-fa86ede4ac2e    RACK1
UN   54.191.72.56    109.54 KB       256     ?               129087eb-9aea-4af6-92c6-99fdadb39c33    RACK1
UN   54.187.25.99    104.94 KB       256     ?               0540c7d7-2622-4f1f-a3f0-acb39282e0fc    RACK1

Datacenter: ASIA-DC
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--   Address         Load            Tokens  Owns            Host ID                                 Rack
UN   54.160.174.243  109.54 KB       256     ?               c7686ffd-7a5b-4124-858e-df2e61130aaa    RACK1
UN   54.235.9.159    109.75 KB       256     ?               39798227-9f6f-4868-8193-08570856c09a    RACK1
UN   54.146.228.25   128.33 KB       256     ?               7a4957a1-9590-4434-9746-9c8a6f796a0c    RACK1
  1. When all nodes are up and running alter the keyspace. Alter the following Keyspaces in the the cluster:
  • Keyspace created by the user (which needed to replicate to the new DC)
  • System: system_auth, system_distributed, system_traces For example, replicate the data to three nodes in the new DC

For example:

Before

ALTER KEYSPACE <keyspace> WITH REPLICATION =
  { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc>:0};

After

ALTER KEYSPACE <keyspace> WITH REPLICATION =
        {'class’: 'NetworkTopologyStrategy', <exiting_dc>:3, <new_dc>:4};
  1. Run nodetool rebuild on each node in the new data-center.

For example:

nodetool rebuild -- <data-center_name>

Procedures