ScyllaDB Documentation Logo Documentation
  • Server
    • ScyllaDB Open Source
    • ScyllaDB Enterprise
    • ScyllaDB Alternator
  • Cloud
  • Tools
    • ScyllaDB Manager
    • ScyllaDB Monitoring Stack
    • ScyllaDB Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
Download
Menu
ScyllaDB Docs ScyllaDB Open Source Scylla for Administrators Upgrade Scylla Upgrade ScyllaDB Open Source Upgrade Guide - ScyllaDB 5.0 to 5.1 Upgrade Guide - ScyllaDB 5.0 to 5.1

Upgrade Guide - ScyllaDB 5.0 to 5.1¶

This document is a step by step procedure for upgrading from ScyllaDB 5.0 to ScyllaDB 5.1, and rollback to version 5.0 if required.

This guide covers upgrading Scylla on Red Hat Enterprise Linux (RHEL) 7/8, CentOS 7/8, Debian 10 and Ubuntu 20.04. It also applies when using ScyllaDB official image on EC2, GCP, or Azure; the image is based on Ubuntu 20.04.

See OS Support by Platform and Version for information about supported versions.

Upgrade Procedure¶

A ScyllaDB upgrade is a rolling procedure which does not require full cluster shutdown. For each of the nodes in the cluster, serially (i.e. one node at a time), you will:

  • Check that the cluster’s schema is synchronized

  • Drain the node and backup the data

  • Backup the configuration file

  • Stop ScyllaDB

  • Download and install new ScyllaDB packages

  • Start ScyllaDB

  • Validate that the upgrade was successful

Apply the following procedure serially on each node. Do not move to the next node before validating that the node you upgraded is up and running the new version.

During the rolling upgrade, it is highly recommended:

  • Not to use the new 5.1 features

  • Not to run administration functions, like repairs, refresh, rebuild or add or remove nodes. See sctool for suspending ScyllaDB Manager (only available for ScyllaDB Enterprise) scheduled or running repairs.

  • Not to apply schema changes

Note

Before upgrading, make sure to use the latest ScyllaDB Monitoring stack.

Upgrade Steps¶

Check the cluster schema¶

Make sure that all nodes have the schema synchronized before upgrade. The upgrade procedure will fail if there is a schema disagreement between nodes.

nodetool describecluster

Drain the nodes and backup the data¶

Before any major procedure, like an upgrade, it is recommended to backup all the data to an external device. In Scylla, backup is done using the nodetool snapshot command. For each node in the cluster, run the following command:

nodetool drain
nodetool snapshot

Take note of the directory name that nodetool gives you, and copy all the directories having that name under /var/lib/scylla to a backup device.

When the upgrade is completed on all nodes, remove the snapshot with the nodetool clearsnapshot -t <snapshot> command to prevent running out of space.

Backup the configuration file¶

sudo cp -a /etc/scylla/scylla.yaml /etc/scylla/scylla.yaml.backup-src

Gracefully stop the node¶

sudo service scylla-server stop

Download and install the new release¶

Before upgrading, check what version you are running now using dpkg -s scylla-server. You should use the same version as this version in case you want to rollback the upgrade. If you are not running a 5.0.x version, stop right here! This guide only covers 5.0.x to 5.1.y upgrades.

To upgrade ScyllaDB:

  1. Update the ScyllaDB deb repo (Debian, Ubuntu) to 5.1.

  2. Install the new ScyllaDB version:

    sudo apt-get clean all
    sudo apt-get update
    sudo apt-get dist-upgrade scylla
    

Answer ‘y’ to the first two questions.

Before upgrading, check what version you are running now using rpm -qa | grep scylla-server. You should use the same version as this version in case you want to rollback the upgrade. If you are not running a 5.0.x version, stop right here! This guide only covers 5.0.x to 5.1.y upgrades.

To upgrade ScyllaDB:

  1. Update the ScyllaDB rpm repo to 5.1.

  2. Install the new ScyllaDB version:

    sudo yum clean all
    sudo yum update scylla\* -y
    

Before upgrading, check what version you are running now using dpkg -s scylla-server. You should use the same version as this version in case you want to rollback the upgrade. If you are not running a 5.0.x version, stop right here! This guide only covers 5.0.x to 5.1.y upgrades.

There are two alternative upgrade procedures:

  • Upgrading ScyllaDB and simultaneously updating 3rd party and OS packages. It is recommended if you are running a ScyllaDB official image (EC2 AMI, GCP, and Azure images), which is based on Ubuntu 20.04.

  • Upgrading ScyllaDB without updating any external packages.

To upgrade ScyllaDB and update 3rd party and OS packages (RECOMMENDED):

Choosing this upgrade procedure allows you to upgrade your ScyllaDB version and update the 3rd party and OS packages using one command.

  1. Update the ScyllaDB deb repo (Debian, Ubuntu) to 5.1.

  2. Load the new repo:

    sudo apt-get update
    
  3. Run the following command to update the manifest file:

    cat scylla-packages-<version>-<arch>.txt | sudo xargs -n1 apt-get install -y
    

    Where:

    • <version> - The ScyllaDB version to which you are upgrading ( 5.1 ).

    • <arch> - Architecture type: x86_64 or aarch64.

    The file is included in the ScyllaDB packages downloaded in the previous step. The file location is http://downloads.scylladb.com/downloads/scylla/aws/manifest/scylla-packages-<version>-<arch>.txt

    Example:

    cat scylla-packages-5.1.2-x86_64.txt | sudo xargs -n1 apt-get install -y
    

    Note

    Alternatively, you can update the manifest file with the following command:

    sudo apt-get install $(awk '{print $1'} scylla-packages-<version>-<arch>.txt) -y

To upgrade ScyllaDB:

  1. Update the ScyllaDB deb repo (Debian, Ubuntu) to 5.1.

  2. Install the new ScyllaDB version:

    sudo apt-get clean all
    sudo apt-get update
    sudo apt-get dist-upgrade scylla
    

Answer ‘y’ to the first two questions.

Start the node¶

sudo service scylla-server start

Validate¶

  1. Check cluster status with nodetool status and make sure all nodes, including the one you just upgraded, are in UN status.

  2. Use curl -X GET "http://localhost:10000/storage_service/scylla_release_version" to check the ScyllaDB version. Validate that the version matches the one you upgraded to.

  3. Check scylla-server log (by journalctl _COMM=scylla) and /var/log/syslog to validate there are no new errors in the log.

  4. Check again after two minutes, to validate no new issues are introduced.

Once you are sure the node upgrade was successful, move to the next node in the cluster.

See ScyllaDB Metrics Update - ScyllaDB 5.0 to 5.1 for more information.

Update the Mode in perftune.yaml¶

Due to performance improvements in version 5.1, your cluster’s existing nodes may use a different mode than the nodes created after the upgrade. Using different modes across one cluster is not recommended, so you should ensure that the same mode is used on all nodes. See Updating the Mode in perftune.yaml After a ScyllaDB Upgrade for instructions.

Rollback Procedure¶

Note

Execute the following commands one node at the time, moving to the next node only after the rollback procedure completed successfully.

The following procedure describes a rollback from ScyllaDB 5.1.x to 5.0.y. Apply this procedure if an upgrade from 5.0 to 5.1 failed before completing on all nodes. Use this procedure only for nodes you upgraded to 5.1.

ScyllaDB rollback is a rolling procedure which does not require full cluster shutdown. For each of the nodes you rollback to 5.0, serially (i.e. one node at a time), you will:

  • Drain the node and stop Scylla

  • Retrieve the old ScyllaDB packages

  • Restore the configuration file

  • Restore system tables

  • Reload systemd configuration

  • Restart ScyllaDB

  • Validate the rollback success

Apply the following procedure serially on each node. Do not move to the next node before validating that the rollback was successful and the node is up and running the old version.

Rollback Steps¶

Drain and gracefully stop the node¶

nodetool drain
sudo service scylla-server stop

Download and install the old release¶

  1. Remove the old repo file.

    sudo rm -rf /etc/apt/sources.list.d/scylla.list
    
  2. Update the ScyllaDB deb repo (Debian, Ubuntu) to 5.0.

  3. Install:

    sudo apt-get update
    sudo apt-get remove scylla\* -y
    sudo apt-get install scylla
    

Answer ‘y’ to the first two questions.

  1. Remove the old repo file.

    sudo rm -rf /etc/yum.repos.d/scylla.repo
    
  2. Update the ScyllaDB rpm repo to 5.0.

  3. Install:

    sudo yum clean all
    sudo rm -rf /var/cache/yum
    sudo yum remove scylla\\*tools-core
    sudo yum downgrade scylla\\* -y
    sudo yum install scylla
    
  1. Remove the old repo file.

    sudo rm -rf /etc/apt/sources.list.d/scylla.list
    
  2. Update the ScyllaDB deb repo (Debian, Ubuntu) to 5.0.

  3. Install:

    sudo apt-get update
    sudo apt-get remove scylla\* -y
    sudo apt-get install scylla
    

Answer ‘y’ to the first two questions.

Restore the configuration file¶

sudo rm -rf /etc/scylla/scylla.yaml
sudo cp -a /etc/scylla/scylla.yaml.backup-src | /etc/scylla/scylla.yaml

Restore system tables¶

Restore all tables of system and system_schema from the previous snapshot because 5.1 uses a different set of system tables. See Restore from a Backup and Incremental Backup for reference.

cd /var/lib/scylla/data/keyspace_name/table_name-UUID/snapshots/<snapshot_name>/
sudo cp -r * /var/lib/scylla/data/keyspace_name/table_name-UUID/
sudo chown -R scylla:scylla /var/lib/scylla/data/keyspace_name/table_name-UUID/

Reload systemd configuration¶

You must reload the unit file if the systemd unit file is changed.

sudo systemctl daemon-reload

Start the node¶

sudo service scylla-server start

Validate¶

Check the upgrade instructions above for validation. Once you are sure the node rollback is successful, move to the next node in the cluster.

PREVIOUS
Upgrade Guide - ScyllaDB 5.0 to 5.1
NEXT
Scylla Metric Update - Scylla 5.0 to 5.1
ScyllaDB Open Source
  • 5.1
    • master
    • 5.2
    • 5.1
  • Getting Started
    • Install Scylla
      • ScyllaDB Web Installer for Linux
      • Scylla Unified Installer (relocatable executable)
      • Air-gapped Server Installation
      • What is in each RPM
      • Scylla Housekeeping and how to disable it
      • Scylla Developer Mode
      • Scylla Configuration Reference
    • Configure Scylla
    • ScyllaDB Requirements
      • System Requirements
      • OS Support by Platform and Version
      • Scylla in a Shared Environment
    • Migrate to ScyllaDB
      • Migration Process from Cassandra to Scylla
      • Scylla and Apache Cassandra Compatibility
      • Migration Tools Overview
    • Integration Solutions
      • Integrate Scylla with Spark
      • Integrate Scylla with KairosDB
      • Integrate Scylla with Presto
      • Integrate Scylla with Elasticsearch
      • Integrate Scylla with Kubernetes
      • Integrate Scylla with the JanusGraph Graph Data System
      • Integrate Scylla with DataDog
      • Integrate Scylla with Kafka
      • Integrate Scylla with IOTA Chronicle
      • Integrate Scylla with Spring
      • Shard-Aware Kafka Connector for Scylla
      • Install Scylla with Ansible
      • Integrate Scylla with Databricks
    • Tutorials
  • Scylla for Administrators
    • Administration Guide
    • Procedures
      • Cluster Management
      • Backup & Restore
      • Change Configuration
      • Maintenance
      • Best Practices
      • Benchmarking Scylla
      • Migrate from Cassandra to Scylla
      • Disable Housekeeping
    • Security
      • Scylla Security Checklist
      • Enable Authentication
      • Enable and Disable Authentication Without Downtime
      • Generate a cqlshrc File
      • Reset Authenticator Password
      • Enable Authorization
      • Grant Authorization CQL Reference
      • Role Based Access Control (RBAC)
      • Scylla Auditing Guide
      • Encryption: Data in Transit Client to Node
      • Encryption: Data in Transit Node to Node
      • Generating a self-signed Certificate Chain Using openssl
      • Encryption at Rest
      • LDAP Authentication
      • LDAP Authorization (Role Management)
    • Admin Tools
      • Nodetool Reference
      • CQLSh
      • REST
      • Tracing
      • Scylla SStable
      • Scylla Types
      • SSTableLoader
      • cassandra-stress
      • SSTabledump
      • SSTable2json
      • SSTable Index
      • Scylla Logs
      • Seastar Perftune
      • Virtual Tables
    • ScyllaDB Monitoring Stack
    • ScyllaDB Operator
    • ScyllaDB Manager
    • Upgrade Procedures
      • Scylla Enterprise
      • Scylla Open Source
      • Scylla Open Source to Scylla Enterprise
      • Scylla AMI
    • System Configuration
      • System Configuration Guide
      • scylla.yaml
      • Scylla Snitches
    • Benchmarking Scylla
  • Scylla for Developers
    • Learn To Use Scylla
      • Scylla University
      • Course catalog
      • Scylla Essentials
      • Basic Data Modeling
      • Advanced Data Modeling
      • MMS - Learn by Example
      • Care-Pet an IoT Use Case and Example
    • Scylla Alternator
    • Scylla Features
      • Scylla Open Source Features
      • Scylla Enterprise Features
    • Scylla Drivers
      • Scylla CQL Drivers
      • Scylla DynamoDB Drivers
  • CQL Reference
    • CQLSh: the CQL shell
    • Appendices
    • Compaction
    • Consistency Levels
    • Consistency Level Calculator
    • Data Definition
    • Data Manipulation
    • Data Types
    • Definitions
    • Global Secondary Indexes
    • Additional Information
    • Expiring Data with Time to Live (TTL)
    • Additional Information
    • Functions
    • JSON Support
    • Materialized Views
    • Non-Reserved CQL Keywords
    • Reserved CQL Keywords
    • ScyllaDB CQL Extensions
  • Scylla Architecture
    • Scylla Ring Architecture
    • Scylla Fault Tolerance
    • Consistency Level Console Demo
    • Scylla Anti-Entropy
      • Scylla Hinted Handoff
      • Scylla Read Repair
      • Scylla Repair
    • SSTable
      • Scylla SSTable - 2.x
      • ScyllaDB SSTable - 3.x
    • Compaction Strategies
    • Raft Consensus Algorithm in ScyllaDB
  • Troubleshooting Scylla
    • Errors and Support
      • Report a Scylla problem
      • Error Messages
      • Change Log Level
    • Scylla Startup
      • Ownership Problems
      • Scylla will not Start
      • Scylla Python Script broken
    • Cluster and Node
      • Failed Decommission Problem
      • Cluster Timeouts
      • Node Joined With No Data
      • SocketTimeoutException
      • NullPointerException
    • Data Modeling
      • Scylla Large Partitions Table
      • Scylla Large Rows and Cells Table
      • Large Partitions Hunting
    • Data Storage and SSTables
      • Space Utilization Increasing
      • Disk Space is not Reclaimed
      • SSTable Corruption Problem
      • Pointless Compactions
      • Limiting Compaction
    • CQL
      • Time Range Query Fails
      • COPY FROM Fails
      • CQL Connection Table
      • Reverse queries fail
    • Scylla Monitor and Manager
      • Manager and Monitoring integration
      • Manager lists healthy nodes as down
  • Knowledge Base
    • Upgrading from experimental CDC
    • Compaction
    • Counting all rows in a table is slow
    • CQL Query Does Not Display Entire Result Set
    • When CQLSh query returns partial results with followed by “More”
    • Run Scylla and supporting services as a custom user:group
    • Decoding Stack Traces
    • Snapshots and Disk Utilization
    • DPDK mode
    • Debug your database with Flame Graphs
    • How to Change gc_grace_seconds for a Table
    • Gossip in Scylla
    • Increase Permission Cache to Avoid Non-paged Queries
    • How does Scylla LWT Differ from Apache Cassandra ?
    • Map CPUs to Scylla Shards
    • Scylla Memory Usage
    • NTP Configuration for Scylla
    • Updating the Mode in perftune.yaml After a ScyllaDB Upgrade
    • POSIX networking for Scylla
    • Scylla consistency quiz for administrators
    • Recreate RAID devices
    • How to Safely Increase the Replication Factor
    • Scylla and Spark integration
    • Increase Scylla resource limits over systemd
    • Scylla Seed Nodes
    • How to Set up a Swap Space
    • Scylla Snapshots
    • Scylla payload sent duplicated static columns
    • Stopping a local repair
    • System Limits
    • How to flush old tombstones from a table
    • Time to Live (TTL) and Compaction
    • Scylla Nodes are Unresponsive
    • Update a Primary Key
    • Using the perf utility with Scylla
    • Configure Scylla Networking with Multiple NIC/IP Combinations
  • ScyllaDB University
  • Scylla FAQ
  • Contribute to ScyllaDB
  • Glossary
  • Alternator: DynamoDB API in Scylla
    • Getting Started With ScyllaDB Alternator
    • Scylla Alternator for DynamoDB users
  • Create an issue
  • Edit this page

On this page

  • Upgrade Guide - ScyllaDB 5.0 to 5.1
    • Upgrade Procedure
    • Upgrade Steps
      • Check the cluster schema
      • Drain the nodes and backup the data
      • Backup the configuration file
      • Gracefully stop the node
      • Download and install the new release
      • Start the node
      • Validate
      • Update the Mode in perftune.yaml
    • Rollback Procedure
    • Rollback Steps
      • Drain and gracefully stop the node
      • Download and install the old release
      • Restore the configuration file
      • Restore system tables
      • Reload systemd configuration
      • Start the node
      • Validate
Logo
Docs Contact Us About Us
Mail List Icon Slack Icon Forum Icon
© 2023, ScyllaDB. All rights reserved.
Last updated on 27 Mar 2023.
Powered by Sphinx 4.3.2 & ScyllaDB Theme 1.3.5