# Adding a New Node Into an Existing ScyllaDB Cluster (Out Scale) When you add a new node, other nodes in the cluster stream data to the new node. This operation is called bootstrapping and may be time-consuming, depending on the data size and network bandwidth. If using a [multi-availability-zone](https://docs.scylladb.com/manual/master/faq.md#faq-best-scenario-node-multi-availability-zone), make sure they are balanced. ## Prerequisites ### Check the Status of Nodes You cannot add new nodes to the cluster if any existing node is down. Before adding new nodes, check the status of the nodes in the cluster using [nodetool status](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/status.md) command. ### Collect Cluster Information Log into one of the nodes in the cluster to collect the following information: > * cluster_name - `grep cluster_name /etc/scylla/scylla.yaml` > * seeds - `grep seeds: /etc/scylla/scylla.yaml` > * endpoint_snitch - `grep endpoint_snitch /etc/scylla/scylla.yaml` > * ScyllaDB version - `scylla --version` > * Authenticator - `grep authenticator /etc/scylla/scylla.yaml` ## Procedure 1. Install ScyllaDB on the nodes you want to add to the cluster. See [Getting Started](https://docs.scylladb.com/manual/master/getting-started/index.md) for further instructions. Follow the ScyllaDB installation procedure up to `scylla.yaml` configuration phase. Make sure that the ScyllaDB version of the new node is identical to the other nodes in the cluster. If the node starts during the process, follow [What to do if a Node Starts Automatically](https://docs.scylladb.com/manual/master/operating-scylla/procedures/cluster-management/clear-data.md). #### NOTE Make sure to use the same ScyllaDB **patch release** on the new/replaced node, to match the rest of the cluster. It is not recommended to add a new node with a different release to the cluster. For example, use the following for installing ScyllaDB patch release (use your deployed version): ```default sudo yum install scylla-2025.1.0 ``` #### NOTE It’s important to keep I/O scheduler configuration in sync on nodes with the same hardware. That’s why we recommend skipping running scylla_io_setup when provisioning a new node with exactly the same hardware setup as existing nodes in the cluster. Instead, we recommend to copy the following files from an existing node to the new node after running scylla_setup and restart scylla-server service (if it is already running): : * /etc/scylla.d/io.conf * /etc/scylla.d/io_properties.yaml Using different I/O scheduler configuration may result in unnecessary bottlenecks. 2. On each node, edit the `scylla.yaml` file `/etc/scylla/` to configure the parameters listed below. > * **cluster_name** - Specifies the name of the cluster. > * **listen_address** - Specifies the IP address that ScyllaDB used to connect to the other ScyllaDB nodes in the cluster. > * **endpoint_snitch** - Specifies the selected snitch. > * **rpc_address** - Specifies the address for CQL client connections. > * **seeds** - Specifies the IP address of an existing node in the cluster. The new node will use this IP to connect to the cluster and learn the cluster topology and state. #### WARNING Adding a node in a new rack in an existing datacenter may violate the [RF-rack-valid](https://docs.scylladb.com/manual/master/reference/glossary.md#term-RF-rack-valid-keyspace) constraints of some keyspace. If a keyspace uses tablets and contains a Materialized View or Secondary Index, or if the rf_rack_valid_keyspaces option is set, the invariant is enforced for the keyspace, and the new node will be rejected if adding it would violate the invariant. 1. Start the nodes with the following command: > Supported OS > ```shell > sudo systemctl start scylla-server > ``` > Docker > ```shell > docker exec -it some-scylla supervisorctl start scylla > ``` > (with *some-scylla* container already running) 2. Verify that the nodes were added to the cluster using [nodetool status](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/status.md) command. Other nodes in the cluster will be streaming data to the new nodes, so the new nodes will be in Up Joining (UJ) status. Wait until the nodes’ status changes to Up Normal (UN) - the time depends on the data size and network bandwidth. **Example:** Nodes in the cluster are streaming data to the new node: ```shell Datacenter: DC1 Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.201 112.82 KB 256 32.7% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1 UN 192.168.1.202 91.11 KB 256 32.9% 125ed9f4-7777-1dbn-mac8-43fddce9123e B1 UJ 192.168.1.203 124.42 KB 256 32.6% 675ed9f4-6564-6dbd-ca08-43fddce952de B1 ``` Nodes in the cluster finished streaming data to the new node: ```shell Datacenter: DC1 Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.201 112.82 KB 256 32.7% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1 UN 192.168.1.202 91.11 KB 256 32.9% 125ed9f4-7777-1dbn-mac8-43fddce9123e B1 UN 192.168.1.203 124.42 KB 256 32.6% 675ed9f4-6564-6dbd-ca08-43fddce952de B1 ``` 3. When the new node status is Up Normal (UN), run the [nodetool cleanup](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/cleanup.md) command on all nodes in the cluster except for the new node that has just been added. Cleanup removes keys that were streamed to the newly added node and are no longer owned by the node. #### NOTE To prevent data resurrection, it’s essential to complete cleanup after adding nodes and before any node is decommissioned or removed. However, cleanup may consume significant resources. Use the following guideline to reduce cleanup impact: Tip 1: When adding multiple nodes, run the cleanup operations after all nodes are added on all nodes but the last one to be added. Tip 2: Postpone cleanup to low demand hours while ensuring it completes successfully before any node is decommissioned or removed. Tip 3: Run cleanup one node at a time, reducing overall cluster impact. 4. Wait until the new node becomes UN (Up Normal) in the output of [nodetool status](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/status.md) on one of the old nodes. 5. If you are using ScyllaDB Monitoring, update the [monitoring stack](https://monitoring.docs.scylladb.com/stable/install/monitoring_stack.html#configure-scylla-nodes-from-files) to monitor it. If you are using ScyllaDB Manager, make sure you install the [Manager Agent](https://manager.docs.scylladb.com/stable/install-scylla-manager-agent.html), and Manager can access it.