Migrate a Keyspace from Vnodes to Tablets¶

Prerequisites¶

Limitations¶

The current migration procedure has the following limitations:

• Clusters with the enforce_rack_list option enabled are not yet supported: the migrated keyspace would keep its numeric replication factors, which the option rejects. Disable the option, run the migration, convert the replication factors to rack lists with ALTER KEYSPACE, then enable it again.

• No schema changes during the migration. Creating or dropping tables, views, or indexes, and dropping the migrating keyspace are rejected until the migration is finished. Altering the properties of existing tables and views is allowed, except enabling CDC, which is not supported.

• No planned topology changes during the migration. Do not add, remove, decommission, or rebuild nodes while a migration is in progress. Replacing a dead node is supported as an emergency recovery operation.

• No repair operations during the migration. Do not run nodetool repair on the migrating keyspace while a migration is in progress.

• No backup and restore for migrating tables during the migration. Do not take snapshots, run backups, or restore into the migrating keyspace while the migration is in progress.

• No TRUNCATE on tables in the migrating keyspace during the migration.

• Only CQL tables, materialized views, and secondary indexes can be migrated. CDC tables and Alternator tables are not supported. Views whose partition key consists of exactly the base table’s partition key columns, in the same order, including the backing views of local secondary indexes, are migrated as co-located tables with their base table (sharing its tablets). Other views, including the backing views of global secondary indexes, get tablets of their own.

• Tables with counters or LWTs cannot be migrated.

• Only tables using the Incremental Compaction Strategy (ICS) can be migrated. Other compaction strategies are not supported.

Overview¶

The migration consists of three phases:

1. Prepare: Create tablet maps for all tables in the keyspace. Each tablet inherits its token range and replica set from the corresponding vnode range.

2. Storage upgrade: Restart each node one at a time, upgrading its storage from vnodes to tablets. Upon restart, the node begins resharding data into tablets. This is a storage-layer operation and is unrelated to ScyllaDB version upgrades.

3. Finalize: Once all nodes have been upgraded, commit the migration by clearing the migration state and switching the keyspace schema to tablets.

During the first two phases, the migration is reversible; you can roll back to vnodes. However, once the migration is finalized, it cannot be reversed.

Procedure¶

1. If the cluster is managed by ScyllaDB Manager, suspend its tasks. Backup, restore, and repair are not yet supported on migrating keyspaces:

   sctool suspend --cluster <cluster-id|name> --no-continue
   

   The --no-continue option deletes the local snapshots of backup tasks. Otherwise, these snapshots would keep a copy of the pre-migration SSTables on disk throughout the migration, as resharding rewrites all SSTables.

2. Prepare the keyspace for migration:

   1. Create tablet maps for all tables in the keyspace:

      nodetool migrate-to-tablets start <keyspace>
      

   2. Verify that the keyspace is in migrating_to_tablets state and all nodes are still using vnodes:

      nodetool migrate-to-tablets status <keyspace>
      

      Example:

      $ nodetool migrate-to-tablets status ks
      Keyspace: ks
      Status: migrating_to_tablets
      
      Nodes:
      Host ID                                Address    Status
      99d8de76-3954-4727-911a-6a07251b180c   10.0.0.1   uses vnodes
      0b5fd6f6-9670-4faf-a480-ad58cf119007   10.0.0.2   uses vnodes
      017dd39a-3d06-4c8a-8ac4-379f9e595607   10.0.0.3   uses vnodes
      

3. Upgrade all nodes to tablets:

   1. Pick a node and open a shell on it to run the following commands.

   2. Mark the node for upgrade to tablets:

      Caution

      Do not mark more than one node for upgrade at the same time. Even if you restart them serially, unexpected restarts can happen for various reasons (crashes, power failures, etc.) leading to parallel node upgrades which can reduce availability.

      nodetool migrate-to-tablets upgrade
      

   3. Verify that the node status changed from vnodes to migrating to tablets:

      nodetool migrate-to-tablets status <keyspace>
      

      Example:

      $ nodetool migrate-to-tablets status ks
      Keyspace: ks
      Status: migrating_to_tablets
      
      Nodes:
      Host ID                                Address    Status
      99d8de76-3954-4727-911a-6a07251b180c   10.0.0.1   migrating to tablets  <---
      0b5fd6f6-9670-4faf-a480-ad58cf119007   10.0.0.2   uses vnodes
      017dd39a-3d06-4c8a-8ac4-379f9e595607   10.0.0.3   uses vnodes
      

   4. Drain and stop the node:

      nodetool drain
      

      Supported OSDocker

      sudo systemctl stop scylla-server
      

   5. Restart the node:

      Supported OSDocker

      sudo systemctl start --no-block scylla-server
      

      The --no-block option makes the command return immediately instead of blocking until resharding completes, so that you can monitor its progress as described in the next step.

   6. Wait until the node is UP and has returned to the ScyllaDB cluster using nodetool status. This operation may take a long time due to resharding. To monitor resharding progress, use the task manager API:

      nodetool tasks list compaction --keyspace <keyspace>
      

      Example:

      $ nodetool tasks list compaction --keyspace ks
      task_id                                type                   kind scope state   sequence_number keyspace table entity shard start_time           end_time
      ccf28999-d19c-4c94-8b61-592a5f10dba8   resharding compaction  node table running 730             ks       t1           0     2026-08-07T08:22:03Z
      

      The command should show one resharding task per migrating table. To inspect the progress of one particular task, pass its task ID to nodetool tasks status and compare the number of resharded bytes (progress_completed) with the total number of bytes to be resharded (progress_total):

      nodetool tasks status <task-id>
      

      Example:

      $ nodetool tasks status ccf28999-d19c-4c94-8b61-592a5f10dba8
      id: ccf28999-d19c-4c94-8b61-592a5f10dba8
      type: resharding compaction
      kind: node
      scope: table
      state: running
      is_abortable: false
      start_time: 2026-08-07T08:22:03Z
      end_time:
      error:
      parent_id: e85de42c-9bf0-3a0b-a3ce-6d481b093bc7
      sequence_number: 730
      shard: 0
      keyspace: ks
      table: t1
      entity:
      progress_units: bytes
      progress_total: 5347926524519
      progress_completed: 252509969840
      children_ids: [{task_id: 22461a60-ae95-4356-a5cb-dceac72c75b5, node: 172.31.0.228 }, {task_id: e2f86734-6213-42b3-85ca-b86864a97dfb, node: 172.31.0.228 }, ...]
      

      Note

      Resharding may be followed by a reshape compaction, which rewrites the resharded SSTables into the layout expected by the table’s compaction strategy. Both resharding and reshaping are part of the node’s boot process and both appear in nodetool tasks list compaction. Wait for both to complete before proceeding to the next step.

   7. Verify that the node status changed from migrating to tablets to uses tablets:

      nodetool migrate-to-tablets status <keyspace>
      

      Example:

      $ nodetool migrate-to-tablets status ks
      Keyspace: ks
      Status: migrating_to_tablets
      
      Nodes:
      Host ID                                Address    Status
      99d8de76-3954-4727-911a-6a07251b180c   10.0.0.1   uses tablets  <---
      0b5fd6f6-9670-4faf-a480-ad58cf119007   10.0.0.2   uses vnodes
      017dd39a-3d06-4c8a-8ac4-379f9e595607   10.0.0.3   uses vnodes
      

   8. Move to the next node and repeat from step a until all nodes are upgraded.

4. Finalize the migration:

   Warning

   Finalization cannot be undone. Once the migration is finalized, the keyspace cannot be switched back to vnodes.

   1. Issue the finalization request:

      nodetool migrate-to-tablets finalize <keyspace>
      

      Finalization switches co-located materialized views and local secondary indexes that use the repair tombstone_gc mode to timeout. After the migration they become co-located tables, which do not support repair, and timeout is their default mode. Other views and the backing views of global secondary indexes keep their mode.

   2. Verify that the keyspace status changed to tablets:

      nodetool migrate-to-tablets status <keyspace>
      

      Example:

      $ nodetool migrate-to-tablets status ks
      Keyspace: ks
      Status: tablets
      

   3. Wait for tablet layout normalization to complete:

      After finalization, a tablet normalization process takes place in the background which transforms the tablets of all migrated tables into a power-of-two layout. Until power-of-two convergence is complete, the tablet layout of all these tables will be suboptimal, causing degraded performance. Run the following command repeatedly to monitor the per-table status of this background work:

      nodetool migrate-to-tablets status <keyspace> --with-tablet-status
      

      Example:

      $ nodetool migrate-to-tablets status ks --with-tablet-status
      Keyspace: ks
      Status: tablets
      
      Tablet info:
        Pow2 tablet layout convergence: in progress
        Tables converging: 2/3
      
      Table   Status       Tablets   Target
      t1      converging   2301      2048
      t2      converging   2176      2048
      t3      converged    2048      -
      

5. If you suspended ScyllaDB Manager in the first step, resume it:

   sctool resume --cluster <cluster-id|name> --no-continue
   

   The --no-continue option makes interrupted tasks start from scratch, instead of continuing from progress recorded before the migration.

6. Run a full cluster repair to ensure that all nodes have the most up-to-date data. As the storage upgrade process kept each node offline for an extended period of time, its dataset might have become stale.

   If the keyspace has materialized views or secondary indexes, the repair also covers updates to them that were lost during finalization. Each node switches to the tablet-based way of pairing base and view replicas when it applies the finalization, so view updates written while the nodes disagreed can miss some view replicas.

   Either run the repair with nodetool, as described below, or let ScyllaDB Manager run it for you.

   For vnode-based keyspaces, run nodetool repair on every node:

   nodetool repair -pr
   

   For tablet-based keyspaces, including the one you just migrated, run nodetool cluster repair on any single node:

   nodetool cluster repair
   

   Run both if the cluster has keyspaces of both kinds.

Rollback Procedure¶

If you need to abort the migration before finalization, you can roll back by downgrading each node back to vnodes. The rollback procedure is the following:

1. Find all nodes that have been upgraded to tablets (status: uses tablets) or they are in the process of upgrading to tablets (status: migrating to tablets):

   nodetool migrate-to-tablets status <keyspace>
   

   Example:

   $ nodetool migrate-to-tablets status ks
   Keyspace: ks
   Status: migrating_to_tablets
   
   Nodes:
   Host ID                                Address    Status
   99d8de76-3954-4727-911a-6a07251b180c   10.0.0.1   uses tablets  <---
   0b5fd6f6-9670-4faf-a480-ad58cf119007   10.0.0.2   migrating to tablets  <---
   017dd39a-3d06-4c8a-8ac4-379f9e595607   10.0.0.3   uses vnodes
   

2. For each upgraded or upgrading node in the cluster, perform a downgrade (one node at a time):

   1. Open a shell on the node to run the following commands.

   2. Mark the node for downgrade:

      nodetool migrate-to-tablets downgrade
      

   3. Check the node status. The status for a previously upgraded node should change from uses tablets to migrating to vnodes. The status for a previously upgrading node should change from migrating to tablets to uses vnodes or migrating to vnodes:

      nodetool migrate-to-tablets status <keyspace>
      

      Example:

      $ nodetool migrate-to-tablets status ks
      Keyspace: ks
      Status: migrating_to_tablets
      
      Nodes:
      Host ID                                Address    Status
      99d8de76-3954-4727-911a-6a07251b180c   10.0.0.1   migrating to vnodes  <---
      0b5fd6f6-9670-4faf-a480-ad58cf119007   10.0.0.2   migrating to tablets
      017dd39a-3d06-4c8a-8ac4-379f9e595607   10.0.0.3   uses vnodes
      

   4. If the node status is uses vnodes, the downgrade is complete. Move to the next node and repeat from step a.

   5. If the node is migrating to vnodes, restart it to complete the downgrade:

      1. Drain and stop the node:

         nodetool drain
         

         Supported OSDocker

         sudo systemctl stop scylla-server
         

      2. Restart the node:

         Supported OSDocker

         sudo systemctl start --no-block scylla-server
         

         The --no-block option makes the command return immediately instead of blocking until resharding completes, so that you can monitor its progress as described in the next step.

      3. Wait until the node is UP and has returned to the ScyllaDB cluster using nodetool status. This operation may take a long time due to resharding. To monitor resharding progress, use the task manager API:

         nodetool tasks list compaction --keyspace <keyspace>
         

         Example:

         $ nodetool tasks list compaction --keyspace ks
         task_id                                type                   kind scope state   sequence_number keyspace table entity shard start_time           end_time
         ccf28999-d19c-4c94-8b61-592a5f10dba8   resharding compaction  node table running 730             ks       t1           0     2026-08-07T08:22:03Z
         

         The command should show one resharding task per migrating table. To inspect the progress of one particular task, pass its task ID to nodetool tasks status and compare the number of resharded bytes (progress_completed) with the total number of bytes to be resharded (progress_total):

         nodetool tasks status <task-id>
         

         Example:

         $ nodetool tasks status ccf28999-d19c-4c94-8b61-592a5f10dba8
         id: ccf28999-d19c-4c94-8b61-592a5f10dba8
         type: resharding compaction
         kind: node
         scope: table
         state: running
         is_abortable: false
         start_time: 2026-08-07T08:22:03Z
         end_time:
         error:
         parent_id: e85de42c-9bf0-3a0b-a3ce-6d481b093bc7
         sequence_number: 730
         shard: 0
         keyspace: ks
         table: t1
         entity:
         progress_units: bytes
         progress_total: 5347926524519
         progress_completed: 252509969840
         children_ids: [{task_id: 22461a60-ae95-4356-a5cb-dceac72c75b5, node: 172.31.0.228 }, {task_id: e2f86734-6213-42b3-85ca-b86864a97dfb, node: 172.31.0.228 }, ...]
         

         Note

         Resharding may be followed by a reshape compaction, which rewrites the resharded SSTables into the layout expected by the table’s compaction strategy. Both resharding and reshaping are part of the node’s boot process and both appear in nodetool tasks list compaction. Wait for both to complete before proceeding to the next node.

      4. Verify that the node status changed from migrating to vnodes to uses vnodes:

         nodetool migrate-to-tablets status <keyspace>
         

         Example:

         $ nodetool migrate-to-tablets status ks
         Keyspace: ks
         Status: migrating_to_tablets
         
         Nodes:
         Host ID                                Address    Status
         99d8de76-3954-4727-911a-6a07251b180c   10.0.0.1   uses vnodes  <---
         0b5fd6f6-9670-4faf-a480-ad58cf119007   10.0.0.2   migrating to tablets
         017dd39a-3d06-4c8a-8ac4-379f9e595607   10.0.0.3   uses vnodes
         

      5. Move to the next node and repeat from step a until all nodes are downgraded.

3. Once all nodes have been downgraded, finalize the rollback:

   nodetool migrate-to-tablets finalize <keyspace>
   

4. If you suspended ScyllaDB Manager before starting the migration, resume it:

   sctool resume --cluster <cluster-id|name> --no-continue
   

   The --no-continue option makes interrupted tasks start from scratch, instead of continuing from progress recorded before the rollback.

5. Run a full cluster repair to ensure that all nodes have the most up-to-date data. As the storage downgrade process kept each node offline for an extended period of time, its dataset might have become stale.

   Either run the repair with nodetool, as described below, or let ScyllaDB Manager run it for you.

   For vnode-based keyspaces, including the one you just rolled back, run nodetool repair on every node:

   nodetool repair -pr
   

   For tablet-based keyspaces, run nodetool cluster repair on any single node:

   nodetool cluster repair
   

   Run both if the cluster has keyspaces of both kinds.

Migrating multiple keyspaces¶

Migrating multiple keyspaces simultaneously is supported. The procedure is the same as with a single keyspace except that the preparation and finalization steps need to be repeated for each keyspace. However, note that a new migration cannot be started once another migration is in the upgrade phase. The migrations need to be prepared and finalized together.

To migrate multiple keyspaces simultaneously, follow these steps:

1. For each keyspace, prepare it for migration:

   nodetool migrate-to-tablets start <keyspace1>
   nodetool migrate-to-tablets start <keyspace2>
   ...
   

   Verify that all keyspaces are in migrating_to_tablets state before proceeding:

   nodetool migrate-to-tablets status <keyspace1>
   nodetool migrate-to-tablets status <keyspace2>
   ...
   

2. Upgrade all nodes in the cluster following the same procedure as for a single keyspace. Each node restart reshards all keyspaces under migration in one pass.

3. For each keyspace, finalize the migration:

   nodetool migrate-to-tablets finalize <keyspace1>
   nodetool migrate-to-tablets finalize <keyspace2>
   ...
   

Replacing a Dead Node during Migration¶

If a node fails during the migration, replace it using the standard procedure for replacing a dead node.

The replacing node must have the same shard count as the replaced node. If the shard count differs, the replacement will fail.

After replacement completes, you can resume the migration. The replacing node always starts in vnode mode and must be upgraded to tablets, regardless of the storage mode of node it replaced.