Upcaling your cluster involves moving the cluster to a larger instance. With this procedure, it can be done without downtime.
Scylla was designed with big servers and multi-cores in mind. In most cases, it is better to run a smaller cluster on a bigger machine instance than a larger cluster on a small machine instance. However, there may be cases where you started with a small cluster, and you now you want to upscale.
There are a few alternatives to do this:
Add Bigger Nodes to a Cluster and removing the old smaller nodes. This is useful when you can not upscale (add more CPU) for each node, for example using I3 instances on EC2.
This procedure can be used to either upscale an entire cluster or to upscale a single node.
Add new bigger nodes to the cluster. Confirm Streaming has completed before continuing.
Remove the old smaller nodes. Confirm Streaming has completed before continuing.
Repeat steps 1 and 2 until the entire cluster is using bigger nodes.
The cluster is only as strong as its weakest node. Do not overload the cluster before all nodes are as upscaled.
This procedure is only useful for entire clusters, not individual nodes. Do the following on each node in the cluster:
Run nodetool drain to stop traffic to the node.
Stop the service
sudo systemctl stop scylla-server
sudo service scylla-server stop
docker exec -it some-scylla supervisorctl stop scylla
(without stopping some-scylla container)
scylla_setup to set Scylla to the new HW configuration.
Start the service
sudo systemctl start scylla-server
sudo service scylla-server start
docker exec -it some-scylla supervisorctl start scylla
(with some-scylla container already running)
Updating the number of cores will cause Scylla to reshard the SSTables to match the new core number. This is done by compacting all of the data on disk at startup.