# Migrate a Keyspace from Vnodes to Tablets Experimental This procedure describes how to migrate an existing keyspace from vnodes to tablets. Tablets are designed to be the long-term replacement for vnodes, offering numerous benefits such as faster topology operations, automatic load balancing, automatic cleanups, and improved streaming performance. Migrating to tablets is strongly recommended. See [Data Distribution with Tablets](https://docs.scylladb.com/manual/master/architecture/tablets.md) for details. ℹ️ This feature is experimental and will change in future releases, including the removal of current limitations. #### NOTE The migration is an online operation. This means that the keyspace remains fully available to users throughout the migration, provided that its replication factor is greater than 1. Reads and writes continue to be served using vnodes until the migration is finished. #### WARNING During the migration, you should expect degraded performance on the migrating keyspace. The reasons are the following: * **Rolling restart**: Each node must upgrade its storage from vnodes to tablets. This is an offline operation happening on startup, so a restart is needed. Upon restart, each node performs a heavy and time-consuming resharding operation to reorganize its data based on tablets, and remains offline until this operation completes. Resharding may last from minutes to hours, depending on the amount of data that the node holds. At this time, the node cannot serve any requests. * **Unbalanced tablets**: The initial tablet layout mirrors the vnode layout. The tablet load balancer does not rebalance tablets until the migration is finished, so some shards may carry more data than others during the migration. The imbalance is expected to be more prominent in clusters with very large nodes (hundreds of vCPUs). * **Loss of shard awareness**: During the migration and until the rolling restart is complete, the cluster is in a mixed state with some nodes using vnodes and others using tablets. In this state, queries may cause cross-shard operations within nodes, reducing performance. The performance will return to normal after the migration finishes and the tablet load balancer rebalances the data. ## Prerequisites ### Cluster state * All nodes in the cluster must be **up and running**. You can check the status of all nodes with [nodetool status](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/status.md). * All nodes must be running ScyllaDB 2026.2 or later. * Disk utilization on every node should be **below 50%**. This headroom helps prevent problems during the migration’s resharding phase, which temporarily requires additional disk space. ### Keyspace and table configuration * The keyspace’s replication strategy must be the **NetworkTopologyStrategy**, and its replication factor must equal the number of racks to ensure that the keyspace will be [RF-rack-valid](https://docs.scylladb.com/manual/master/reference/glossary.md#term-RF-rack-valid-keyspace) after the migration. If your configuration requires changes, see [Update Topology Strategy From Simple to Network](https://docs.scylladb.com/manual/master/operating-scylla/procedures/cluster-management/update-topology-strategy-from-simple-to-network.md) and [How to Safely Increase the Replication Factor](https://docs.scylladb.com/manual/master/kb/rf-increase.md). * All tables in the keyspace should have `tombstone_gc` mode set to `repair` to prevent tombstone garbage collection from running during migration. For guidance, see [Tombstone Garbage Collection](https://docs.scylladb.com/manual/master/cql/ddl.md#ddl-tombstones-gc). ## Limitations The current migration procedure has the following limitations: * **Multi-DC clusters** are supported, but every datacenter in the migrating keyspace’s replication strategy must have a replication factor of at least one. Datacenters with a replication factor of zero (RF=0) are not yet supported. * Clusters with [zero-token nodes](https://docs.scylladb.com/manual/master/architecture/zero-token-nodes.md) are not yet supported. * **No schema changes** during the migration. Do not create, alter, or drop tables in the migrating keyspace until the migration is finished. * **No topology changes** during the migration. Do not add, remove, decommission, replace, or rebuild nodes while a migration is in progress. * **No repair** operations during the migration. Do not run `nodetool repair` on the migrating keyspace while a migration is in progress. * **No TRUNCATE** on tables in the migrating keyspace during the migration. * Only **CQL base tables** can be migrated. Materialized views, secondary indexes, CDC tables, and Alternator tables are not supported. * Tables with **counters** or **LWTs** cannot be migrated. * Only tables using the **Incremental Compaction Strategy (ICS)** can be migrated. Other compaction strategies are not supported. ## Overview The migration consists of three phases: 1. **Prepare**: Create tablet maps for all tables in the keyspace. Each tablet inherits its token range and replica set from the corresponding vnode range. 2. **Storage upgrade**: Restart each node one at a time, upgrading its storage from vnodes to tablets. Upon restart, the node begins resharding data into tablets. This is a storage-layer operation and is unrelated to ScyllaDB version upgrades. 3. **Finalize**: Once all nodes have been upgraded, commit the migration by clearing the migration state and switching the keyspace schema to tablets. During the first two phases, the migration is reversible; you can roll back to vnodes. However, once the migration is finalized, it cannot be reversed. #### NOTE In the following sections, any reference to “upgrade” or “downgrade” of a node will refer to the migration of its storage from vnodes to tablets or vice versa. Do not confuse it with version upgrades/downgrades. ## Procedure 1. Prepare the keyspace for migration: 1. Create tablet maps for all tables in the keyspace: ```console scylla nodetool migrate-to-tablets start ``` 2. Verify that the keyspace is in `migrating_to_tablets` state and all nodes are still using vnodes: ```console scylla nodetool migrate-to-tablets status ``` **Example:** ```console $ scylla nodetool migrate-to-tablets status ks Keyspace: ks Status: migrating_to_tablets Nodes: Host ID Status 99d8de76-3954-4727-911a-6a07251b180c uses vnodes 0b5fd6f6-9670-4faf-a480-ad58cf119007 uses vnodes 017dd39a-3d06-4c8a-8ac4-379f9e595607 uses vnodes ``` 2. Upgrade all nodes to tablets: 1. Pick a node. 2. Mark the node for upgrade to tablets: #### NOTE This is a node-local operation. Use the IP address of the node that you are upgrading. ```console scylla nodetool -h migrate-to-tablets upgrade ``` 3. Verify that the node status changed from `vnodes` to `migrating to tablets`: ```console scylla nodetool migrate-to-tablets status ``` **Example:** ```console $ scylla nodetool migrate-to-tablets status ks Keyspace: ks Status: migrating_to_tablets Nodes: Host ID Status 99d8de76-3954-4727-911a-6a07251b180c migrating to tablets <--- 0b5fd6f6-9670-4faf-a480-ad58cf119007 uses vnodes 017dd39a-3d06-4c8a-8ac4-379f9e595607 uses vnodes ``` 4. Drain and stop the node: ```console scylla nodetool -h drain ``` Supported OS ```shell sudo systemctl stop scylla-server ``` Docker ```shell docker exec -it some-scylla supervisorctl stop scylla ``` (without stopping *some-scylla* container) 5. Restart the node: Supported OS ```shell sudo systemctl start scylla-server ``` Docker ```shell docker exec -it some-scylla supervisorctl start scylla ``` (with *some-scylla* container already running) 6. Wait until the node is UP and has returned to the ScyllaDB cluster using [nodetool status](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/status.md). This operation may take a long time due to resharding. To monitor resharding progress, use the task manager API: ```console scylla nodetool tasks list compaction -h --keyspace | grep -i reshard ``` 7. Verify that the node status changed from `migrating to tablets` to `uses tablets`: ```console scylla nodetool migrate-to-tablets status ``` **Example:** ```console $ scylla nodetool migrate-to-tablets status ks Keyspace: ks Status: migrating_to_tablets Nodes: Host ID Status 99d8de76-3954-4727-911a-6a07251b180c uses tablets <--- 0b5fd6f6-9670-4faf-a480-ad58cf119007 uses vnodes 017dd39a-3d06-4c8a-8ac4-379f9e595607 uses vnodes ``` 8. Move to the next node and repeat from step a until all nodes are upgraded. 3. Finalize the migration: #### WARNING Finalization **cannot be undone**. Once the migration is finalized, the keyspace cannot be switched back to vnodes. 1. Issue the finalization request: ```console scylla nodetool migrate-to-tablets finalize ``` 2. Verify that the keyspace status changed to `tablets`: ```console scylla nodetool migrate-to-tablets status ``` **Example:** ```console $ scylla nodetool migrate-to-tablets finalize ks Keyspace: ks Status: tablets ``` 3. Optionally, wait for tablet layout normalization to complete: After finalization, a tablet normalization process takes place in the background which transforms the tablets of all migrated tables into a power-of-two layout. Until power-of-two convergence is complete, the tablet layout of all these tables will be suboptimal, causing degraded performance. Run the following command repeatedly to monitor the per-table status of this background work: ```console scylla nodetool migrate-to-tablets status --with-tablet-status ``` **Example:** ```console $ scylla nodetool migrate-to-tablets status ks --with-tablet-status Keyspace: ks Status: tablets Tablet info: Pow2 tablet layout convergence: in progress Tables converging: 2/3 Table Status Tablets Target t1 converging 2301 2048 t2 converging 2176 2048 t3 converged 2048 - ``` ## Rollback Procedure #### NOTE Rollback is only possible **before finalization**. Once the migration is finalized, it cannot be reversed. If you need to abort the migration **before finalization**, you can roll back by downgrading each node back to vnodes. The rollback procedure is the following: 1. Find all nodes that have been upgraded to tablets (status: `uses tablets`) or they are in the process of upgrading to tablets (status: `migrating to tablets`): ```console scylla nodetool migrate-to-tablets status ``` **Example:** ```console $ scylla nodetool migrate-to-tablets status ks Keyspace: ks Status: migrating_to_tablets Nodes: Host ID Status 99d8de76-3954-4727-911a-6a07251b180c uses tablets <--- 0b5fd6f6-9670-4faf-a480-ad58cf119007 migrating to tablets <--- 017dd39a-3d06-4c8a-8ac4-379f9e595607 uses vnodes ``` 2. For **each upgraded or upgrading node** in the cluster, perform a downgrade (one node at a time): 1. Mark the node for downgrade: ```console scylla nodetool -h migrate-to-tablets downgrade ``` 2. Check the node status. The status for a previously upgraded node should change from `uses tablets` to `migrating to vnodes`. The status for a previously upgrading node should change from `migrating to tablets` to `uses vnodes` or `migrating to vnodes`: ```console scylla nodetool migrate-to-tablets status ``` **Example:** ```console $ scylla nodetool migrate-to-tablets status ks Keyspace: ks Status: migrating_to_tablets Nodes: Host ID Status 99d8de76-3954-4727-911a-6a07251b180c migrating to vnodes <--- 0b5fd6f6-9670-4faf-a480-ad58cf119007 migrating to tablets 017dd39a-3d06-4c8a-8ac4-379f9e595607 uses vnodes ``` 3. If the node status is `uses vnodes`, the downgrade is complete. Move to the next node and repeat from step a. 4. If the node is `migrating to vnodes`, restart it to complete the downgrade: 1. Drain and stop the node: ```console scylla nodetool -h drain ``` Supported OS ```shell sudo systemctl stop scylla-server ``` Docker ```shell docker exec -it some-scylla supervisorctl stop scylla ``` (without stopping *some-scylla* container) 2. Restart the node: Supported OS ```shell sudo systemctl start scylla-server ``` Docker ```shell docker exec -it some-scylla supervisorctl start scylla ``` (with *some-scylla* container already running) 3. Wait until the node is UP and has returned to the ScyllaDB cluster using [nodetool status](https://docs.scylladb.com/manual/master/operating-scylla/nodetool-commands/status.md). This operation may take a long time due to resharding. To monitor resharding progress, use the task manager API: ```console scylla nodetool tasks list compaction -h --keyspace | grep -i reshard ``` 4. Verify that the node status changed from `migrating to vnodes` to `uses vnodes`: ```console scylla nodetool migrate-to-tablets status ``` **Example:** ```console $ scylla nodetool migrate-to-tablets status ks Keyspace: ks Status: migrating_to_tablets Nodes: Host ID Status 99d8de76-3954-4727-911a-6a07251b180c uses vnodes <--- 0b5fd6f6-9670-4faf-a480-ad58cf119007 migrating to tablets 017dd39a-3d06-4c8a-8ac4-379f9e595607 uses vnodes ``` 5. Move to the next node and repeat from step a until all nodes are downgraded. 3. Once all nodes have been downgraded, finalize the rollback: ```console scylla nodetool migrate-to-tablets finalize ``` ## Migrating multiple keyspaces Migrating multiple keyspaces simultaneously is supported. The procedure is the same as with a single keyspace except that the preparation and finalization steps need to be repeated for each keyspace. However, note that a new migration cannot be started once another migration is in the upgrade phase. The migrations need to be prepared and finalized together. To migrate multiple keyspaces simultaneously, follow these steps: 1. For **each keyspace**, prepare it for migration: ```console scylla nodetool migrate-to-tablets start scylla nodetool migrate-to-tablets start ... ``` Verify that all keyspaces are in `migrating_to_tablets` state before proceeding: ```console scylla nodetool migrate-to-tablets status scylla nodetool migrate-to-tablets status ... ``` 2. Upgrade all nodes in the cluster following the same [procedure](#upgrade-nodes) as for a single keyspace. Each node restart reshards all keyspaces under migration in one pass. 3. For **each keyspace**, finalize the migration: ```console scylla nodetool migrate-to-tablets finalize scylla nodetool migrate-to-tablets finalize ... ```