# Remove a Data-Center from a ScyllaDB Cluster Before removing (decommission) the data-center from the cluster, it is essential to verify that the data-center that is about to be removed does not hold any unique data. ## Prerequisites * Verify that there are no client writes to nodes in the data-center that is to be decommissioned. * Check the cluster status using the [nodetool status](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/status.md) command to understand the cluster deployment. ```shell Datacenter: US-DC Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.201 112.82 KB 256 32.7% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1 UN 192.168.1.202 91.11 KB 256 32.9% 125ed9f4-7777-1dbn-lac8-23fddce9123e B1 UN 192.168.1.203 124.42 KB 256 32.6% 675ed9f4-6564-6dbd-ca08-43fddce952de B1 Datacenter: ASIA-DC Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 50.191.1.204 112.82 KB 256 32.7% 4d5ed9f4-7764-4ded-dad8-63fdace94b7c B1 UN 50.191.1.205 91.11 KB 256 32.9% 145id9f4-7777-1dvn-nac8-83fdzce917r4 B1 UN 50.191.1.206 124.42 KB 256 32.6% 777ed9f4-6564-6dsd-can8-13fdxce999gy B1 Datacenter: EUROPE-DC Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.91.202.31 112.82 KB 256 32.7% 1d5ed8f4-7764-4dbd-rad8-44fddce94b7v B1 UN 172.91.202.32 91.11 KB 256 32.9% 525ed7g4-7437-1dbn-mac8-53fddce9123c B1 UN 172.91.202.33 124.42 KB 256 32.6% 975edbm4-6564-63bd-san8-73fddce952ga B1 ``` ## Procedure 1. If there are vnode keyspaces in this DC, run the `nodetool repair -pr` command on each node in the data-center that is going to be decommissioned. This will verify that all the data is in sync between the decommissioned data-center and the other data-centers in the cluster. For example: If the ASIA-DC cluster is to be removed, then, run the `nodetool repair -pr` command on all the nodes in the ASIA-DC 2. If there are tablet keyspaces in this DC, run the `nodetool cluster repair` on an arbitrary node. The reason for running repair is to ensure that any updates stored only on the about-to-be-decommissioned replicas are propagated to the other replicas, before the replicas on the decommissioned datacenter are dropped. 3. ALTER every cluster KEYSPACE, so that the keyspaces will no longer replicate data to the decommissioned data-center. For example: ```shell cqlsh> DESCRIBE cqlsh> CREATE KEYSPACE WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', '' : 3, '' : 3, '' : 3}; cqlsh> ALTER KEYSPACE WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', '' : 3, '' : 3, '' : 0}; ``` For example: There are three data-centers: US-DC, ASIA-DC and EUROPE-DC, the ASIA-DC is to be removed. The current KEYSPACE is: ```shell cqlsh> DESCRIBE nba cqlsh> CREATE KEYSPACE nba WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 2, 'EUROPE-DC' : 3}; ``` Use the ALTER KEYSPACE to change the KEYSPACE and remove the decommissioned data-center. ```shell cqlsh> ALTER KEYSPACE nba WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 0, 'EUROPE-DC' : 3}; ``` For tablet keyspaces, update the replication factor one by one: ```shell cqlsh> DESCRIBE nba2 cqlsh> CREATE KEYSPACE nba2 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 2, 'EUROPE-DC' : 3} AND tablets = { 'enabled': true }; ``` ```shell cqlsh> ALTER KEYSPACE nba2 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 1, 'EUROPE-DC' : 3} AND tablets = { 'enabled': true }; cqlsh> ALTER KEYSPACE nba2 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 0, 'EUROPE-DC' : 3} AND tablets = { 'enabled': true }; ``` #### NOTE If `rf_rack_valid_keyspaces` option is set, a tablet keyspace needs to use rack list replication factor, so that the DC can be removed. See [the conversion procedure](https://docs.scylladb.com/manual/master/cql/ddl.md#conversion-to-rack-list-rf). In this case, to remove a datacenter: ```shell cqlsh> DESCRIBE nba3 cqlsh> CREATE KEYSPACE nba3 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : ['RAC4', 'RAC5'], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8']} AND tablets = { 'enabled': true }; ``` ```shell cqlsh> ALTER KEYSPACE nba3 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : ['RAC4'], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8']} AND tablets = { 'enabled': true }; cqlsh> ALTER KEYSPACE nba3 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : [], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8']} AND tablets = { 'enabled': true }; ``` Consider [upgrading rf_rack_valid_keyspaces option to enforce_rack_list option](https://docs.scylladb.com/manual/master/architecture/tablets.md#keyspace-rf-rack-valid-to-enforce-rack-list) to ensure all tablet keyspaces use rack lists. If the keyspace uses rack list replication, update the replication factor in one `ALTER KEYSPACE` statement, under the following rules: : * Existing datacenters must keep their current replication factor. * An existing datacenter can be removed (**N to 0**). * A new datacenter can be assigned a replication factor (**0 to N**). #### WARNING While removing a datacenter and altering keyspaces, do **not** perform any reads or writes that involve the datacenter being removed. In particular, avoid using global consistency levels (such as `ALL`, `EACH_QUORUM`) that would include the decommissioned datacenter in the operation. Use `LOCAL_*` consistency levels (e.g., `LOCAL_QUORUM`, `LOCAL_ONE`) until the datacenter is fully decommissioned. ```shell cqlsh> DESCRIBE nba4 cqlsh> CREATE KEYSPACE nba4 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : ['RAC4', 'RAC5'], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8']} AND tablets = { 'enabled': true }; ``` The following is **not** allowed because it changes the replication factor of `EUROPE-DC` (adds `RAC9`) and removes `ASIA-DC` in the same statement: ```shell cqlsh> ALTER KEYSPACE nba4 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : [], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8', 'RAC9']} AND tablets = { 'enabled': true }; ``` Remove all replicas from the decommissioned datacenter: ```shell cqlsh> ALTER KEYSPACE nba4 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : [], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8']} AND tablets = { 'enabled': true }; ``` #### NOTE If table audit is enabled, the `audit` keyspace is automatically created with `NetworkTopologyStrategy`. You must also alter the `audit` keyspace to remove replicas from the decommissioned data-center. For example: ```shell cqlsh> ALTER KEYSPACE audit WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 0, 'EUROPE-DC' : 3}; ``` Failure to do so will result in decommission errors such as “zero replica after the removal”. #### WARNING Removal of replicas from a datacenter cannot be aborted. To get back to the previous replication, wait until the ALTER KEYSPACE finishes and then add the replicas back by running another ALTER KEYSPACE statement. 4. Run [nodetool decommission](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/decommission.md) on every node in the data center that is to be removed. Refer to [Remove a Node from a ScyllaDB Cluster - Down Scale](https://docs.scylladb.com/manual/master/operating-scylla/procedures/cluster-management/remove-node.md) for further information. For example: If ASIA-DC is to be removed, then, execute the `nodetool decommission` command on all the nodes in this data-center. 5. Verify that the data-center was successfully removed by using the [nodetool status](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/status.md) command. For example: ```shell Datacenter: US-DC Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.201 112.82 KB 256 32.7% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1 UN 192.168.1.202 91.11 KB 256 32.9% 125ed9f4-7777-1dbn-mac8-43fddce9123e B1 UN 192.168.1.203 124.42 KB 256 32.6% 675ed9f4-6564-6dbd-ca08-43fddce952de B1 Datacenter: EUROPE-DC Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.91.202.31 112.82 KB 256 32.7% 1d5ed8f4-7764-4dbd-rad8-44fddce94b7v B1 UN 172.91.202.32 91.11 KB 256 32.9% 525ed7g4-7437-1dbn-mac8-53fddce9123c B1 UN 172.91.202.33 124.42 KB 256 32.6% 975edbm4-6564-63bd-san8-73fddce952ga B1 ```