Replace a Running Node in a Scylla Cluster

There are two methods to replace a running node in a Scylla cluster

  1. Add a new node to the cluster and then decommission the old node
  2. Replace a running node - by taking it place in the cluster

Add a new node to the cluster and then decommission the old node

Adding a new node and only then decommissioning the second node allows the cluster to keep the same level of data replication throughout the process, but with a cost. When adding a new node to a Scylla cluster, existing nodes will give the new node responsibility for a subset of their vNodes, making sure that data is once again equally distributed. In the process, these nodes will stream relevant data to the new node. When decommissioning a node from a Scylla cluster, it will give its vNodes to others nodes, making sure data is once again equally distributed. In the process, this node will stream its data to the other nodes. Hance, replacing a node by adding and decommissioning redistribute the vNodes twice, streaming a node worth of data each time

Procedure

  1. Follow the Adding a New Node Into an Existing Scylla Cluster procedure
  2. Decommission the old node using the Remove a Node from a Scylla Cluster procedure
  3. Run the nodetool cleanup command on all the remaining nodes in the cluster
  4. Verify that the node removed using nodetool status command

Replace a running node - by taking it place in the cluster

Stopping a node and taking its place in the cluster is not as safe, as data replication is reduced during the process. However, it is more efficient, as vNode distribution does not change, and only one node worth of data is streamed

Procedure

1. Stop the Scylla node you want to replace

sudo systemctl stop scylla-server

  1. Follow the Replace a Dead Node in a Scylla Cluster procedure
  2. Verify that the node is successfully replaced using nodetool status command

Procedures