Add a Decommissioned Node Back to a Scylla Cluster

This procedure describes how to add a node to a Scylla cluster after it was decommissioned. On some cases, one would like to add a decommissioned node back to the cluster, for example, if the node was decommissioned by mistake. The following procedure describes the procedure of doing that, by clearing all data from it, and adding it as a new node in the cluster

Prerequisites

Before adding the new node again to the cluster, we need to verify that the node was removed using the nodetool status command

For Example:

Decommissioned node IP is 192.168.1.203, we can see that this node is not in the cluster

Datacenter: DC1
  Status=Up/Down
  State=Normal/Leaving/Joining/Moving
  --  Address        Load       Tokens  Owns (effective)                         Host ID         Rack
  UN  192.168.1.201  112.82 KB  256     32.7%             8d5ed9f4-7764-4dbd-bad8-43fddce94b7c   B1
  UN  192.168.1.202  91.11 KB   256     32.9%             125ed9f4-7777-1dbn-mac8-43fddce9123e   B1

Procedure

1. Stop the decommissioned node using

CentOS, RHEL or Ubuntu 16.04

sudo systemctl stop scylla-server

Ubuntu 14.04 or Debian

sudo service scylla-server stop

Docker (without stopping some-scylla container)

docker exec -it some-scylla supervisorctl stop scylla
2. Delete the data and commitlog folders
sudo rm -rf /var/lib/scylla/data
sudo find /var/lib/scylla/commitlog -type f -delete
sudo find /var/lib/scylla/hints -type f -delete
sudo find /var/lib/scylla/view_hints -type f -delete

Since the node is added back to the cluster as a new node, deleting the node data folder is a must. Otherwise, old node state, like bootstrap status, will prevent the new node to start the init procedure

3. Follow the Adding a New Node Into an Existing Scylla Cluster procedure to add the decommissioned node back into the cluster