Apache Cassandra to Scylla Migration Process

Migrating data from Apache Cassandra to an eventually consistent data store such as Scylla for a high volume, low latency application and verifying its consistency, is a multi-step process. It involves the following high level steps:

  1. Creating the same schema from Apache Cassandra in Scylla, though there can be some variation
  2. Configuring your application/s to perform dual writes (still reading only from Apache Cassandra)
  3. Taking a snapshot of all to-be-migrated data from Apache Cassandra
  4. Loading the sstable files to Scylla using the Scylla sstableloader tool + Data validation
  5. Verification period: dual writes and reads, Scylla serves reads. Logging mismatches, until minimal data mismatch threshold is reached
  6. Apache Cassandra End Of Life: Scylla only for reads and writes

Note: steps 2 and 5 are required for Live migration only (meaning with on-going traffic and no downtime).

../../_images/cassandra-to-scylla-1.png

Dual Writes: Application logic is updated to write to both DBs

../../_images/cassandra-to-scylla-2.png

Forklifting: Migrate historical data from Apache Cassandra SSTables to Scylla

../../_images/cassandra-to-scylla-3.png

Dual Reads: Ongoing validation of data sync between the two DBs

../../_images/live_migration_timeline.PNG

Live Migration: Migrating from DB-OLD to DB-NEW timeline

Procedure

  1. Create manually / Migrate your schema (keyspaces, tables and user defined type, if used) on / to your Scylla cluster. When migrating from Apache Cassandra 3.x some schema updates are required (see limitations and known issues section).

    • Export schema from Apache Cassandra: cqlsh [IP] "-e DESC SCHEMA" > orig_schema.cql
    • Import schema to Scylla: cqlsh [IP] --file 'adjusted_schema.cql'
  2. If you wish to perform the migration process without any downtime, please configure your application/s to perform dual writes, to both data stores, Apache Cassandra and Scylla (see below code snippet for dual writes). Before doing that, and as a general guidance make sure to use client generated timestamp (writetime). If you do not, the data on Scylla and Apache Cassandra can be considered different, while it is really the same.

    Note: your application/s should continue reading and writing from Apache Cassandra until the entire migration process is completed, data integrity validated and dual writes and reads verification period performed to your satisfaction.

Dual writes and client generated timestamp Python code snippet

# put both writes (cluster 1 and cluster 2) into a list
    writes = []
    #insert 1st statement into db1 session, table 1
    writes.append(db1.execute_async(insert_statement_prepared[0], values))
    #insert 2nd statement into db2 session, table 2
    writes.append(db2.execute_async(insert_statement_prepared[1], values))

    # loop over futures and output success/fail
    results = []
    for i in range(0,len(writes)):
        try:
            row = writes[i].result()
            results.append(1)
        except Exception:
            results.append(0)
            #log exception if you like
            #logging.exception('Failed write: %s', ('Cluster 1' if (i==0) else 'Cluster 2'))

    results.append(values)
    log(results)

    #did we have failures?
    if (results[0]==0):
        #do something, like re-write to cluster 1
        log('Write to cluster 1 failed')
    if (results[1]==0):
        #do something, like re-write to cluster 2
        log('Write to cluster 2 failed')

for x in range(0,RANDOM_WRITES):
    #explicitly set a writetime in microseconds
    values = [ random.randrange(0,1000) , str(uuid.uuid4()) , int(time.time()*1000000) ]

    execute( values )

See the full code example here

  1. On each Apache Cassandra node, take a snapshot for every keyspace using nodetool snapshot command. This will flush all sstables to disk and generate a snapshots folder with an epoch timestamp for each underlying table in that keyspace.

    Folder path post snapshot: /var/lib/cassandra/data/keyspace/table-[uuid]/snapshots/[epoch_timestamp]/

  2. We strongly advise against running the sstableloader tool directly on the Scylla cluster, as it will consume resources from Scylla, instead you should run the sstableloader from intermediate node/s. In order to do that, you need to install scylla-tools-core package (it includes the sstableloader tool).

    You need to make sure you have connectivity to both the Apache Cassandra and Scylla clusters. There are 2 ways to do that, both require having a file system in place (RAID is optional):

    • Option 1 (recommended): copy the sstable files from the Apache Cassandra cluster to a local folder on the intermediate node.
    • Option 2: NFS mount point on the intermediate node to the sstable files located in the Apache Cassandra nodes.
    1. After installing the relevant pkgs (detailed in the links), edit /etc/exports file on each Apache Cassandra node and add the following in a single line:

      [Full path to snapshot ‘epoch’ folder] [Scylla_IP](rw,sync,no_root_squash,no_subtree_check)

    2. Restart NFS server sudo systemctl restart nfs-kernel-server

    3. Create a new folder on one of the Scylla nodes and use it as a mount point to the Apache Cassandra node

      Example:

      sudo mount [Cassandra_IP]:[Full path to snapshots ‘epoch’ folder] /[ks]/[table]

      Note: both the local folder, or the NFS mount point paths must end with /[ks]/[table] format, used by the sstableloader for parsing purposes (see sstableloader help for more details).

  3. If you cannot use intermediate node/s (see previous step), then you have two options:

    • Option 1: Copy the sstable files to a local folder on one of your Scylla cluster nodes. Preferably on a disk or disk-array which is not part of the Scylla cluster RAID, yet still accessible for the sstableloader tool.

      Note: copying it to the Scylla RAID, will require sufficient disk space (Apache Cassandra sstable snapshots size x2 < 50% of Scylla node capacity), in order to contain both the copied sstables files and the entire data migrated to Scylla (keyspace RF should also be taken into account).

    • Option 2: NFS mount point on Scylla nodes to the sstable files located in the Apache Cassandra nodes (see NFS mount instructions in previous step). This saves the additional disk space needed in the 1st option.

      Note: both the local folder or the NFS mount point paths must end with /[ks]/[table] format, used by the sstableloader for parsing purposes (see sstableloader help for more details).

  4. Use Scylla sstableloader tool (NOT the Apache Cassandra one which has the same name) to load the sstables. Running without any parameters will present the list of options and usage. Most important are the sstables directory and the Scylla node IP.

    Note: we recommend using the -x flag for prepared statements, as it boosts the performance (up to x5 factor).

    Examples:

    • sstableloader -x -d [Scylla IP] .../[ks]/[table]
    • sstableloader -x -d [scylla IP] .../[mount point] (in /[ks]/[table] format)
  5. We recommend running several sstableloaders in parallel and utilizing all Scylla nodes as targets for sstable loading. Start with one keyspace and its underlying sstable files from all Apache Cassandra nodes. After completion, continue to the next keyspace and so on.

    Note: limit the sstableloader speed by using the throttling -t parameter, considering your physical HW, live traffic load and network utilization (see sstableloader help for more details).

  6. Once you completed loading the sstable files from all keyspaces, you can use cqlsh or any other tool to validate the data migrated successfully. We strongly recommend configuring your application to perform both writes and reads to/from both data stores. Apache Cassandra (as is, up to this point) and Scylla (now as primary) for a verification period. Keep track of the number of requests for which the data in both these data stores mismatched.

  7. Apache Cassandra end of life: once you are confident in your Scylla cluster, you can flip the flag in your application/s, stop writes and reads against the Cassandra cluster, and make Scylla your sole target/source.

Failure Handling

What should I do if sstableloader fails?

Each loading job is per keyspace/table_name, that means in any case of failure, you need to repeat the loading job again. As you are loading the same data (partially loaded before the failure), compactions will take care of any duplication.

What should I do if an Apache Cassandra node fails?

If the node that failed was a node you were loading sstables from, then the sstableloader will also fail. If you were using RF>1 then the data exists on other node/s, hence you can continue with the sstable loading from all the other Cassandra nodes. Once completed, all your data should be on Scylla.

What should I do if a Scylla node fails?

If the node that failed was a node you were loading sstables to, then the sstableloader will also fail. Restart the loading job and use a different Scylla node as your target.

How to rollback and start from scratch?

  1. Stop the dual writes to Scylla
  2. Stop Scylla service sudo systemctl stop scylla-server
  3. Use cqlsh to perform truncate on all data already loaded to Scylla
  4. Start the dual writes again to Scylla
  5. Take a new snapshot on all Cassandra nodes
  6. Start loading sstables again to Scylla from the NEW snapshot folder

Limitations and Known Issues

  1. Duration data type is not supported in Scylla v1.7 (issue-2240). This is relevant only when migrating from Apache Cassandra 3.X.

  2. Changes in table schema from Apache Cassandra 3.0 that requires adjustments for Scylla 1.7 table schema:

  3. Scylla v1.7 CQL client cqlsh does not displays the millisecond values of a timestamp data type.

    Workaround: Use Cassandra CQL client cqlsh, it will require installing the relevant Cassandra version python driver first sudo pip install cassandra-driver

  4. Nodetool tablestats partition keys (estimated) number in Scylla, post migration from Apache Cassandra, differs by 20% less up to 120% more than the original amount in Cassandra (issue-2545)

Schema differences between Apache Cassandra 3.x and Scylla 1.7.x

The following table illustrates the default schema differences between Apache Cassandra 3.x and Scylla 1.7.x.

Notable differences:

  • ‘caching’ section is supported in Scylla, yet requires adjustments to the schema (see below).
  • ‘crc_check_chance’ (marked in bold) is NOT supported in Scylla, make sure you remove it from the schema.
Cassandra 3.10 (uses 3.x Schema) Scylla 1.7.x (uses 2.1 Schema)
CREATE KEYSPACE mykeyspace WITH replication =
{'class': 'SimpleStrategy', 'replication_factor': '1'}
AND durable_writes = true;

CREATE TYPE mykeyspace.udt_info (
birthday date,
nationality text,
height int);

CREATE TABLE
mykeyspace.all_types_no_counter_no_duration (
aascii ascii,
abigint bigint,
ablob blob,
aboolean boolean,
adate date,
adecimal decimal,
adouble double,
afloat float,
afrozen_udt frozen<udt_info>,
ainet inet,
aint int,
alist list<int>,
amap map<int, int>,
aset set<int>,
asmallint smallint,
atext text,
atime time,
atimestamp timestamp,
atimeuuid timeuuid,
atinyint tinyint,
atuple frozen<tuple<int, text>>,
auuid uuid,
avarchar text,
avarint varint,
PRIMARY KEY (aascii, abigint))
WITH CLUSTERING ORDER BY (abigint ASC)
AND bloom_filter_fp_chance = 0.01
AND comment = ''
CREATE KEYSPACE mykeyspace WITH replication =
{'class': 'SimpleStrategy', 'replication_factor': '1'}
AND durable_writes = true;

CREATE TYPE mykeyspace.udt_info (
birthday date,
nationality text,
height int);

CREATE TABLE
mykeyspace.all_types_no_counter_no_duration (
aascii ascii,
abigint bigint,
ablob blob,
aboolean boolean,
adate date,
adecimal decimal,
adouble double,
afloat float,
afrozen_udt frozen<udt_info>,
ainet inet,
aint int,
alist list<int>,
amap map<int, int>,
aset set<int>,
asmallint smallint,
atext text,
atime time,
atimestamp timestamp,
atimeuuid timeuuid,
atinyint tinyint,
atuple frozen<tuple<int, text>>,
auuid uuid,
avarchar text,
avarint varint,
PRIMARY KEY (aascii, abigint))
WITH CLUSTERING ORDER BY (abigint ASC)
AND bloom_filter_fp_chance = 0.01
AND comment = ''
AND caching = {'keys': 'ALL','rows_per_partition': 'NONE'}
AND compaction = {'class': 'org.apache.cassandra.db.
compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64','class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND speculative_retry = '99PERCENTILE'

(Adjust for 2.1 schema)

AND caching = '{"keys":"ALL","rows_per_partition":"ALL"}'
AND compaction = {'class': 'SizeTieredCompactionStrategy'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND speculative_retry = '99.0PERCENTILE'
AND crc_check_chance = 1.0 (Remove from 2.1 schema)
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0;
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_perios_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0;

© 2016, The Apache Software Foundation.

Apache®, Apache Cassandra®, Cassandra®, the Apache feather logo and the Apache Cassandra® Eye logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.