Safely Remove a Joining Node

Sometimes when adding a node to the cluster it gets stuck in JOINING state (UJ) and never completes the process to a Up-Normal Status (UN). The only solution is to remove the node. As long as the node did not join the cluster (meaning it never went into UN state), you can just stop this node, and clean its data, and try again.

  1. Run the nodetool drain command (Scylla stops listening to its connections from the client and other nodes).
  2. Stop the node

CentOS, RHEL or Ubuntu 16.04

sudo systemctl stop scylla-server

Ubuntu 14.04 or Debian

sudo service scylla-server stop

Docker (without stopping some-scylla container)

docker exec -it some-scylla supervisorctl stop scylla
  1. Clean the data
sudo rm -rf /var/lib/scylla/data
sudo find /var/lib/scylla/commitlog -type f -delete
sudo find /var/lib/scylla/hints -type f -delete
sudo find /var/lib/scylla/view_hints -type f -delete
  1. Start the node

CentOS, RHEL or Ubuntu 16.04

sudo systemctl start scylla-server

Ubuntu 14.04 or Debian

sudo service scylla-server start

Docker (with some-scylla container already running)

docker exec -it some-scylla supervisorctl start scylla