Restore from a Backup and Incremental Backup

Restoring a keyspace from a backup requires all snapshot files of the tables, and (if available) incremental backup files taken after the snapshot. Before restoring from backup, the table data must be truncated, making sure that the existing data does not overwrite the restored data.

Note

For cluster wide backup and restore, see Scylla Manager

Procedure

From one of the nodes, recreate the schema.

cqlsh -e "SOURCE '/path_to_schema/<schema_name.cql>'"

For example:

cqlsh -e "SOURCE 'centos/db_schema.cql'"

Repeat the following steps for each node in the cluster:

  1. Run the nodetool drain command to ensure the data is flushed to the SSTables

  2. Shut down the node

    sudo systemctl stop scylla-server
    
    sudo service scylla-server stop
    
    docker exec -it some-scylla supervisorctl stop scylla
    

    (without stopping some-scylla container)

  3. Delete all the files in the commitlog. Deleting the commitlog will prevent the newer insert from overriding the restored data.

    sudo rm -rf /var/lib/scylla/commitlog/*

  4. Delete all the files in the keyspace_name_table. Note that by default the snapshots are created under Scylla data directory /var/lib/scylla/data/keyspace_name/table_name-UUID/.

    Make sure NOT to delete the existing snapshots in the process.

    For example:

    sudo ll /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000
    
    -rw-r--r-- 1 scylla   scylla     66 Mar  5 09:19 nba-team_players-ka-1-CompressionInfo.db
    -rw-r--r-- 1 scylla   scylla    669 Mar  5 09:19 nba-team_players-ka-1-Data.db
    -rw-r--r-- 4 scylla   scylla     10 Mar  5 08:46 nba-team_players-ka-1-Digest.sha1
    -rw-r--r-- 1 scylla   scylla     24 Mar  5 09:19 nba-team_players-ka-1-Filter.db
    -rw-r--r-- 1 scylla   scylla    218 Mar  5 09:19 nba-team_players-ka-1-Index.db
    -rw-r--r-- 1 scylla   scylla     38 Mar  5 09:19 nba-team_players-ka-1-Scylla.db
    -rw-r--r-- 1 scylla   scylla   4446 Mar  5 09:19 nba-team_players-ka-1-Statistics.db
    -rw-r--r-- 1 scylla   scylla     89 Mar  5 09:19 nba-team_players-ka-1-Summary.db
    -rw-r--r-- 4 scylla   scylla    101 Mar  5 08:46 nba-team_players-ka-1-TOC.txt
    drwx------ 5 scylla   scylla     69 Mar  6 08:14 snapshots
    drwx------ 2 scylla   scylla      6 Mar  5 08:40 upload
    
    sudo rm -f  /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000/*
    
    rm: cannot remove ‘/var/lib/scylla/data/nba/team_roster-c019f8108fda11e8b16a000000000001/snapshots’: Is a directory
    rm: cannot remove ‘/var/lib/scylla/data/nba/team_roster-c019f8108fda11e8b16a000000000001/upload’: Is a directory
    
    sudo ll /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000/
    
    drwx------ 5 scylla   scylla     69 Mar  6 08:14 snapshots
    drwx------ 2 scylla   scylla      6 Mar  5 08:40 upload
    
  5. Select the snapshot you want to restore (usually the most recent one)

    /var/lib/scylla/data/keyspace_name/table_name-UUID/snapshots/<snapshot_name>
    

    For example:

    cd /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000/snapshots/1487847672222
    
  6. Copy the snapshots directory content to the /var/lib/scylla/data/keyspace_name/table_name-UUID/ directory

    For example:

    sudo cp -r * /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000
    
  7. If you have incremental backup files, copy them from the backups folder /var/lib/scylla/data/keyspace_name/table_name-UUID/backups/<backups_name> to the /var/lib/scylla/data/keyspace_name/table_name-UUID/ directory

    For example:

    sudo cp -r * /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000
    
  8. Make sure that all files are owned by the scylla user and group:

    sudo chown -R scylla:scylla /var/lib/scylla/data/mykeyspace/team_players-6e856600017f11e790f4000000000000
    
  9. Start the node

    sudo systemctl start scylla-server
    
    sudo service scylla-server start
    
    docker exec -it some-scylla supervisorctl start scylla
    

    (with some-scylla container already running)

  10. Run nodetool repair command to guarantee that your data is consistent with other nodes.

After performing the above on all nodes, repair the cluster with nodetool repair. This makes sure that the data is consistent on all nodes and between each node.