Handling Node Failures¶

Examples¶

Manual Recovery Procedure¶

You can follow the manual recovery procedure when the majority of nodes (for example, 2 out of 3) failed and are irrecoverable.

During the manual recovery procedure you’ll restart live nodes in a special recovery mode, which will cause the cluster to initialize the Raft algorithm from scratch. However, this time, faulty nodes will not participate in the algorithm. Then, you will replace all faulty nodes (using the standard node replacement procedure). Finally, you will leave the recovery mode and remove the obsolete internal Raft data.

Prerequisites

• Before proceeding, make sure that the irrecoverable nodes are truly dead, and not, for example, temporarily partitioned away due to a network failure. If it is possible for the ‘dead’ nodes to come back to life, they might communicate and interfere with the recovery procedure and cause unpredictable problems.

  If you have no means of ensuring that these irrecoverable nodes won’t come back to life and communicate with the rest of the cluster, setup firewall rules or otherwise isolate your alive nodes to reject any communication attempts from these dead nodes.

• Ensure all live nodes are in the normal state using nodetool status. If there is a node that is joining or leaving, it cannot be recovered. You must permanently stop it. After performing the recovery procedure, use nodetool status ony any other node. If the stopped node appears in the output, it means that other nodes still consider it a member of the cluster, and you should remove it with the node removal procedure.

• Check if the cluster lost data. If the number of dead nodes is equal or larger than your keyspaces RF, then some of the data is lost, and you need to retrieve it from backup. After completing the manual recovery procedure restore the data from backup.

• Decide whether to shut down your service for the manual recovery procedure. ScyllaDB serves data queries during the procedure, however, you may not want to rely on it if:

  • you lost some data, or

  • restarting a single node could lead to unavailability of data queries (the procedure involves a rolling restart). For example, if you are using the standard RF=3, CL=QUORUM setup, you have two datacenters, all nodes in one of the datacenters are dead and one node in the other datacenter is dead, restarting another node in the other datacenter will cause temporary data query unavailability (until the node finishes restarting).

Procedure

1. Perform a rolling restart of your live nodes.

2. Find the group 0 ID by performing the following query on any live node, using e.g. cqlsh:

   cqlsh> SELECT value FROM system.scylla_local WHERE key = 'raft_group0_id';
   

   The group 0 ID is needed in the following steps.

3. Find commit_idx of all live nodes by performing the following query on every live node:

   cqlsh> SELECT commit_idx FROM system.raft WHERE group_id = <group 0 ID>;
   

   Choose a node with the largest commit_idx. If there are multiple such nodes, choose any of them. The chosen node will be the recovery leader.

4. Perform the following queries on every live node:

   cqlsh> TRUNCATE TABLE system.discovery;
   cqlsh> DELETE value FROM system.scylla_local WHERE key = 'raft_group0_id';
   

5. Perform a rolling restart of all live nodes, but:

   • restart the recovery leader first,

   • before restarting each node, add the recovery_leader property to its scylla.yaml file and set it to the host ID of the recovery leader,

   • after restarting each node, make sure it participated in Raft recovery; look for one of the following messages in its logs:

   storage_service - Performing Raft-based recovery procedure with recovery leader <host ID of the recovery leader>/<IP address of the recovery leader>
   storage_service - Raft-based recovery procedure - found group 0 with ID <ID of the new group 0; different from the one used in other steps>
   

   After completing this step, Raft should be fully functional.

6. Replace all dead nodes in the cluster using the node replacement procedure.

   Note

   Removing some of the dead nodes with the node removal procedure is also possible, but it may require decreasing RF of your keyspaces. With tablets enabled, nodetool removenode is rejected if there are not enough nodes to satisfy RF of any tablet keyspace in the node’s datacenter.

7. Remove the recovery_leader property from the scylla.yaml file on all nodes. Send the SIGHUP signal to all ScyllaDB processes to ensure the change is applied.

8. Perform the following queries on every live node:

   cqlsh> DELETE FROM system.raft WHERE group_id = <group 0 ID>;
   cqlsh> DELETE FROM system.raft_snapshots WHERE group_id = <group 0 ID>;
   cqlsh> DELETE FROM system.raft_snapshot_config WHERE group_id = <group 0 ID>;