Upgrade Guide - Ubuntu 14.04 to 16.04¶
Ubuntu 14.04 will be end of life in April 2019, this document is a step by step procedure for Scylla users to upgrade Ubuntu 14.04 to 16.04.
Execute the following commands one node at the time, moving to the next node only after the upgrade procedure completed successfully.
The upgrade is a rolling procedure which does not require full cluster shutdown. For each of the nodes in the cluster, serially (i.e. one at a time), you will:
- Check cluster schema
- Drain node and backup the data
- Backup configuration file
- Stop Scylla
- Upgrade OS to latest 14.04
- Clean scylla-gcc73 packages to avoid conflict
- Upgrade OS to 16.04
- Upgrade Scylla packages
- Start Scylla
- Validate that the upgrade was successful
Apply the following procedure serially on each node. Do not move to the next node before validating the node is up and running with the new version.
During the rolling upgrade it is highly recommended:
- Not to use new 3.0 features
- Not to run administration functions, like repairs, refresh, rebuild or add or remove nodes
- Not to apply schema changes
Make sure that all nodes have the schema synched prior to upgrade, we won’t survive an upgrade that has schema disagreement between nodes.
Before any major procedure, like an upgrade, it is recommended to backup all the data to an external device. In Scylla, backup is done using the
nodetool snapshot command. For each node in the cluster, run the following command:
nodetool drain nodetool snapshot
Take note of the directory name that nodetool gives you, and copy all the directories having this name under
/var/lib/scylla to a backup device.
When the upgrade is complete (all nodes), the snapshot should be removed by
nodetool clearsnapshot -t <snapshot>, or you risk running out of space.
for conf in $(cat /var/lib/dpkg/info/scylla-*server.conffiles /var/lib/dpkg/info/scylla-*conf.conffiles /var/lib/dpkg/info/scylla-*jmx.conffiles | grep -v init ); do sudo cp -v $conf $conf.backup-2.3; done
Related issue: https://github.com/scylladb/scylla/issues/4164
sudo apt-get remove scylla-gcc73-libstdc++6
This step will take some time and will require manual interaction.
sudo apt-get update sudo do-release-upgrade sudo reboot
Update scylla repo to repo for Ubuntu 16.04 - Scylla RPM repo
sudo apt-get dist-upgrade scylla
- Check cluster status with
nodetool statusand make sure all nodes, including the one you just upgraded, are in UN status.
curl -X GET "http://localhost:10000/storage_service/scylla_release_version"to check scylla version.
- Check scylla-server log (execute
journalctl _COMM=scyllafor Ubuntu 16.04) and
/var/log/syslogto validate there are no errors.
- Check again after 2 minutes, to validate no new issues are introduced.
Once you are sure the node upgrade is successful, move to the next node in the cluster.