Upgrade Guide - Ubuntu 14.04 to 16.04¶
Ubuntu 14.04 will be end of life in April 2019, this document is a step by step procedure for Scylla users to upgrade Ubuntu 14.04 to 16.04.
OS: Ubuntu 14.04
Scylla: 2.3, 3.0
Execute the following commands one node at the time, moving to the next node only after the upgrade procedure completed successfully.
If you are using CDC and upgrading Scylla 4.3 to 4.4, please review the API updates in querying CDC streams and CDC stream generations. In particular, you should update applications that use CDC according to CDC Upgrade notes before upgrading the cluster to 4.4.
If you are using CDC and upgrading from pre 4.3 version to 4.3, note the upgrading from experimental CDC.
The upgrade is a rolling procedure which does not require full cluster shutdown. For each of the nodes in the cluster, serially (i.e. one at a time), you will:
Check cluster schema
Drain node and backup the data
Backup configuration file
Upgrade OS to latest 14.04
Clean scylla-gcc73 packages to avoid conflict
Upgrade OS to 16.04
Upgrade Scylla packages
Validate that the upgrade was successful
Apply the following procedure serially on each node. Do not move to the next node before validating the node is up and running with the new version.
During the rolling upgrade it is highly recommended:
Not to use new 3.0 features
Not to run administration functions, like repairs, refresh, rebuild or add or remove nodes
Not to apply schema changes
Make sure that all nodes have the schema synched prior to upgrade, we won’t survive an upgrade that has schema disagreement between nodes.
Before any major procedure, like an upgrade, it is recommended to backup all the data to an external device. In Scylla, backup is done using the
nodetool snapshot command. For each node in the cluster, run the following command:
nodetool drain nodetool snapshot
Take note of the directory name that nodetool gives you, and copy all the directories having this name under
/var/lib/scylla to a backup device.
When the upgrade is complete (all nodes), the snapshot should be removed by
nodetool clearsnapshot -t <snapshot>, or you risk running out of space.
for conf in $(cat /var/lib/dpkg/info/scylla-*server.conffiles /var/lib/dpkg/info/scylla-*conf.conffiles /var/lib/dpkg/info/scylla-*jmx.conffiles | grep -v init ); do sudo cp -v $conf $conf.backup-2.3; done
sudo service scylla-server stop
sudo apt-get clean all sudo apt-get update sudo apt-get upgrade
Related issue: https://github.com/scylladb/scylla/issues/4164
sudo apt-get remove scylla-gcc73-libstdc++6
This step will take some time and will require manual interaction.
sudo apt-get update sudo do-release-upgrade sudo reboot
Update scylla repo to repo for Ubuntu 16.04 - Scylla RPM repo
sudo apt-get dist-upgrade scylla
sudo systemctl start scylla-server
Check cluster status with
nodetool statusand make sure all nodes, including the one you just upgraded, are in UN status.
curl -X GET "http://localhost:10000/storage_service/scylla_release_version"to check scylla version.
Check scylla-server log (execute
journalctl _COMM=scyllafor Ubuntu 16.04) and
/var/log/syslogto validate there are no errors.
Check again after 2 minutes, to validate no new issues are introduced.
Once you are sure the node upgrade is successful, move to the next node in the cluster.