Restore from a Backup and Incremental Backup¶

Choosing a restore method¶

ScyllaDB supports several restore methods. Choose the one that matches where your backup files are and whether the cluster topology changed since the backup was taken:

• Restore from object storage - restore SSTables backed up to S3-compatible object storage with nodetool backup. Runs on a running cluster and works regardless of cluster topology changes since the backup.

• Restore with load and stream - upload backed-up SSTable files to a running cluster; the data is streamed to the nodes owning it. Works regardless of cluster topology changes since the backup.

• Restore to an identical cluster - copy snapshot files back in place and restart the nodes. Requires a cluster with the same number of nodes and the same token distribution as at the time of the backup, and each node must be restored from the backup of the same node. Suitable for vnode-based keyspaces only.

For cluster-wide backup and restore, use ScyllaDB Manager, which orchestrates the process across the cluster.

Prerequisites¶

The following steps and notes apply to all restore methods.

cqlsh -e "SOURCE '/path_to_schema/<schema_name.cql>'"

cqlsh -e "SOURCE 'centos/db_schema.cql'"

cqlsh -e "TRUNCATE <keyspace_name>.<table_name>"

cqlsh -e "TRUNCATE mykeyspace.team_players"

Restore from object storage¶

Use this method to restore SSTables backed up to S3-compatible object storage with nodetool backup. The node downloads the SSTables from the bucket and streams their contents to the nodes owning the data (load and stream), so the restore works regardless of cluster topology changes since the backup, and the cluster stays online.

The object storage endpoint must be configured on the nodes, as described in Configuring Object Storage.

Procedure

1. Complete the prerequisites.

2. List the backed-up SSTables in the bucket under the prefix used during the backup. The restore command takes the paths of the TOC.txt components of the SSTables to restore, relative to the prefix – the remainder of each object key after the prefix. Note that listing tools print full object keys, from the bucket root, so the prefix needs to be stripped. For example:

   aws s3 ls --recursive s3://bucket-foo/ks/cf/24601/ | awk '/-TOC.txt$/ { print $4 }' | sed 's|^ks/cf/24601/||'
   

3. Run nodetool restore, passing the endpoint, bucket, prefix, target keyspace and table, and the list of prefix-relative TOC paths:

   nodetool restore --endpoint s3.us-east-2.amazonaws.com --bucket bucket-foo --prefix ks/cf/24601 \
     --keyspace ks --table cf \
     me-3gdq_0bki_2dy4w2gqj6hoso4mw1-big-TOC.txt \
     me-3gdq_0bki_2dipc1ysb2x2a3btgh-big-TOC.txt
   

   Alternatively, put the same prefix-relative TOC paths (newline-separated) in a file and pass it with the --sstables-file-list option:

   cat > sstables.list <<EOF
   me-3gdq_0bki_2dy4w2gqj6hoso4mw1-big-TOC.txt
   me-3gdq_0bki_2dipc1ysb2x2a3btgh-big-TOC.txt
   EOF
   
   nodetool restore --endpoint s3.us-east-2.amazonaws.com --bucket bucket-foo --prefix ks/cf/24601 \
     --keyspace ks --table cf --sstables-file-list sstables.list
   

4. Monitor the restore. By default, the command waits for the restore to finish and reports its final status. With the --nowait option, it returns a task ID immediately; use the nodetool tasks commands to track progress or cancel the operation.

Speeding up the restore

A single nodetool restore invocation runs on one node, which downloads and streams all the listed SSTables. To parallelize the work, split the list of SSTables between the nodes and run nodetool restore on each of them. The --scope option (node, rack, dc, or all) constrains where each node streams the data, so that concurrent restores don’t stream the same partition to a replica more than once. See nodetool restore for details on combining --scope with per-node SSTable lists.

With the --primary-replica-only option, each partition is streamed only to its primary replica. This reduces the amount of streamed data, but you must run a full cluster repair after the restore completes to replicate the data to the remaining replicas: for vnode-based keyspaces, run nodetool repair -pr on every node; for tablet-based keyspaces, run nodetool cluster repair on any single node.

Restore with load and stream¶

Use this method when the backed-up SSTable files are available on disk (for example, snapshot files copied back from external storage). The SSTables are read and their contents are streamed to the nodes owning the data, so the method works regardless of cluster topology changes since the backup. Each SSTable needs to be uploaded to only one node, any node, and the cluster stays online.

Procedure

1. Complete the prerequisites.

2. Copy the backed-up SSTable files of a table to that table’s upload directory on one of the nodes, and make sure the files are owned by the scylla user and group:

   sudo cp /path/to/backup/sstables/* /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000/upload/
   sudo chown -R scylla:scylla /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000/upload/
   

   You can distribute the backup files between several nodes to parallelize the restore; make sure each SSTable is uploaded to only one node.

3. Run nodetool refresh with the --load-and-stream option on each node holding uploaded files:

   nodetool refresh mykeyspace team_players --load-and-stream
   

   See Load and Stream for the --scope and --primary-replica-only options that constrain the set of target replicas. If --primary-replica-only is used, run a full cluster repair after the restore completes to replicate the data to the remaining replicas: for vnode-based keyspaces, run nodetool repair -pr on every node; for tablet-based keyspaces, run nodetool cluster repair on any single node.

Restore to an identical cluster¶

This method places the snapshot files directly back into the table directories and restarts the nodes.

Complete the prerequisites first.