Database Monsters of the World Connect! Virtual Conference | January 12-14
Register Now

Replacing a dead seed node

In Scylla, it is not possible to bootstrap a seed node. The following steps describe how to replace a dead seed node.

Prerequisites

Verify that the node is a seed node. Use the command below. cat /etc/scylla/scylla.yaml | grep seeds: If the dead node’s IP address is in the seeds list, it needs to be replaced.

Procedure

  • Perform steps 1-3 for all the nodes in the cluster:

  1. Promote an existing node from the cluster to be a seed node by adding the node IP to the seed list in the scylla.yaml file. It can be found under /etc/scylla/

  2. Remove the dead node IP from the seeds providers list as explained in removing a seed node

  3. Restart the node in the cluster using

sudo systemctl restart scylla-server
sudo service scylla-server restart
docker exec -it some-scylla supervisorctl restart scylla

(without restarting some-scylla container)

Note

Since this operation needs to be performed on all nodes in the cluster, it is necessary to orchestrate the procedure such that only a small number of nodes are restarted simultaneously. Use nodetool status to verify restarted nodes are online before restarting more nodes. If too many nodes are offline the cluster may suffer temporary service degradation or outage.

It is recommended to have more than one seed node in your cluster (it’s not allowed to define all the nodes in the cluster as a seed node).