Remove a Node from a Scylla Cluster (Down Scale)¶
Use these instructions when you want to remove nodes in-order to reduce the size of your cluster.
- Run the nodetool status command to check the status of the nodes in your cluster
Datacenter: DC1 Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.201 112.82 KB 256 32.7% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1 UN 192.168.1.202 91.11 KB 256 32.9% 125ed9f4-7777-1dbn-mac8-43fddce9123e B1 DN 192.168.1.203 124.42 KB 256 32.6% 675ed9f4-6564-6dbd-can8-43fddce952gy B1
- If the node status is Up Normal (UN), use the
nodetool decommission guarantees that the node streams its own data to the remaining nodes, so there is no possibility
of data loss. Aside from hard node failures,
nodetool decommission is always preferred.
Review current disk space utilization on existing nodes and make sure the amount of data streamed from the node being removed can fit into the disk space available on the remaining nodes. If there is not enough disk space on the remaining nodes, the removal of a node will fail. Add more storage to remaining nodes before starting the removal procedure.
nodetool netstats command to monitor the progress of the token reallocation.
- If the node status is Down Normal (DN) and cannot be restored, use the
Attention When using
nodetool removenode, the data in the node is not streamed. For this reason,
in order to be able to keep quorum consistencies it is extremely advisable to run a full cluster repair before
nodetool removenode so all existing replicas have the most up to date data.
nodetool removenode we need to use the Host ID of the node
nodetool removenode 675ed9f4-6564-6dbd-can8-43fddce952gy
nodetool netstats command to monitor the progress of the token reallocation
- If the node status is Up Joining (UJ) and does not chenge to Up Normal, see this procedure on how to remove the node.
- Verify that the node removed by using the
Datacenter: DC1 Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.201 112.82 KB 256 32.7% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1 UN 192.168.1.202 91.11 KB 256 32.9% 125ed9f4-7777-1dbn-mac8-43fddce9123e B1
6. When a node was removed from the cluster his data doesn’t removed automatically. The data and commitlog stored on that node need to be removed manually. If you do not do this the old data will still be counted against the load on that node.
To delete the data use
sudo rm -rf /var/lib/scylla/data sudo find /var/lib/scylla/commitlog -type f -delete sudo find /var/lib/scylla/hints -type f -delete sudo find /var/lib/scylla/view_hints -type f -delete