Scylla FAQ

Performance

I have an 8 core server and top shows Scylla is using 800% cpu! What’s up with that?

Starting from scylla 0.14 Scylla will NOT use 100% CPU when idle, only when serving requests so this shouldn’t happen. Scylla does have an non default –poll-mode option in which it is waiting for an event, such as network packet arrival, or a message from another core, is surprisingly time-consuming. To improve efficiency, Scylla uses polling to look for such events, instead of requesting the operating system to dispatch those events. This causes Scylla to consume all available CPU, but improves throughput and latency.

I’m not getting the level of performance I expected. What’s wrong?

It is time to put the tracing suite on. Please read performance tracing Scylla and follow up. Another case is that your data or log files are probably not on an XFS-formatted partition. For best results, data and log files must reside on an XFS-formatted partition. The main issue to figure out is where is the bottleneck. It can be in the client or caused by network/disk/cpu/other.

Scylla is using all of my memory! Why is that? What if the server runs out of memory?

Scylla uses available memory to cache your data. Scylla knows how to dynamically manage memory for optimal performance; for example, if many clients connect to Scylla, it will evict some data from the cache to make room for these connections; when the connection count drops again, this memory is returned to the cache.

Can I limit Scylla to use less CPU and memory?

The --smp option (for instance, --smp 2) will restrict Scylla to a smaller number of CPUs. It will still use 100 % of those CPUs, but at least won’t take your system out completely. An analogous option exists for memory: -m.

Do I ever need to disable the Scylla cache to use less memory?

It is not possible to turn off the Scylla cache. But cache problems do not arise in normal operation. Scylla can use up to 50% of memory for cache, and will dynamically evict rows from the cache if they are too large. So the only possibility of getting an out of memory error is if a single row is bigger than 50% of the memory for a shard. This is (total_machine_memory / (number_of_lcores * 2)).

For a 64GB machine with 16 cores and hyperthreading enabled, you have 2GB per shard, of which the cache can use 1GB per shard. With such large rows you will have other problems. We recommend staying with rows that are less than a few megabytes maximum size.

What are some of the techniques Scylla uses to achieve its performance?

Scylla tries to utilize all available resources (processor cores, memory, storage, and networking) by always operations in parallel and never blocking. If Scylla needs to read a disk block, it initiates the read and immediately moves on to another task. Later, when the read completes Scylla resumes the original task from where it left of. By never blocking, a high degree of concurrency is achieved, allowing all resources to be utilized to their limit.

I understood Scylla underline Seastar framework use one thread per core, but I see more than two threads per core

Seastar creates an extra thread per core for blocking syscalls (like open()/ fsync() / close() ); this allows the Seastar reactor to continue executing while a blocking operation takes place. Those threads are usually idle, so they don’t contribute to significant context switching activity.

I’m seeing X compaction running in parallel on a single Scylla node. Is it normal?

Yes, for more than one reason:

  • each shard (core) will run its compactions independently, often at the same time
  • each table will run its compactions independently, often at the same time
  • depending on the compaction strategy, more than one compaction can run in parallel. For example in Sized Tier Compaction Strategy (STCS), large sstable compaction can take time, allowing smaller sstable to be compacted at the same time

Features

Will Scylla have a certain feature in an upcoming release?

Check the Status page for features scheduled for our GA release.

Which version(s) of Cassandra is Scylla compatible with? Will Scylla be compatible with future Cassandra versions?

Check the Cassandra Compatibility section for current and future Cassandra release compatibility.

We plan to support the 2.2 series features and 3.0 as well, including materialized views.

Beyond 3.0, we will look at the features and see if they make sense for Scylla, but the general plan is to keep compatibility.

We plan to introduce extensions and enhancements. We’ll try to work with the Cassandra community on new features. so that we don’t fragment the ecosystem, and will support switching back and forth between the two implementations while using the same drivers and application code.

Ubuntu

Check and update Ubuntu 14.04 kernel

Running Scylla on Ubuntu 14.04 requires kernel 3.15 or later

  • To check your kernel version: $ uname -a
  • If your kernel is older than 3.15 then:
    • Check for available kernels: $ sudo apt-cache search linux-image
    • Install: $ sudo apt-get install linux-image-your_version_choice, for example linux-image-3.16.0
    • restart: $ sudo reboot now

Installation

Can I install Scylla on a Cassandra server?

Scylla comes with its own version of the Cassandra client tools, in the package scylla-tools. Trying to install it on a server with Cassandra already installed may result in something like:

Unpacking scylla-tools (1.0.1-20160411.b9fe89b-ubuntu1) ...
dpkg: error processing archive /var/cache/apt/archives/scylla-tools_1.0.1-20160411.b9fe89b-ubuntu1_all.deb (--unpack):
trying to overwrite '/usr/bin/nodetool', which is also in package cassandra 2.1.4

We recommend uninstalling Cassandra before installing scylla-tools.

Which snitch or replication strategy should I use?

If you are creating a production cluster or if your cluster is going to have more than one data center you need to use a DC-aware snitch, e.g. GossipingPropertyFileSnitch or Ec2MutliRegionSnitch. You will also need to use a DC-aware replication strategy, e.g. NetworkTopologyStrategy.

However if you are going to create your first cluster or want to try something simple, if your cluster is going to have a single data center then you may use a SimpleSnitch and then use a SimpleStrategy for your keyspaces.

Our general recommendation is to always use a NetworkTopologyStrategy and use Ec2XXX snitches on AWS based clusters and GossipingPropertyFileSnitch in all other cases.

A description of all snitch options we support may be found here: Snitches.

Note: trying to mix a SimpleSnitch with a DC-aware strategies or a DC-aware snitch with a SimpleStrategy may cause your cluster not to work as intended therefore we strongly discourage these types of configurations in general.

Not using a proper snitch-strategy combination may cause different types of errors.

For instance:

Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level for cl LOCAL_ONE. Requires 1, alive 0" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'LOCAL_ONE'}

while all Nodes are alive and are shown in a nodetool status report.

If you see this error you should always check that you are not using a SimpleSnitch in your cluster configuration in conjunction with some DC-aware replication strategy for a keyspace of a table you are failing to query.

The problem may also arise if you are using some DC-aware snitch, e.g. Ec2MultiRegionSnitch, and a SimpleStrategy in a multi-DC cluster.

Please, make sure that both a snitch and a replication strategy of a keyspace are either both of a Simple kind or both are DC-aware.

After that, if you are using a DC-aware configuration, make sure that the replication strategy uses the proper data centers names. Verify the data centers names in your cluster using a nodetool status command.

More info

Where can I ask a question not covered here?

Two mailing lists.

  • scylladb-dev: Discuss the development of Scylla itself.
  • scylladb-users: Discuss using Scylla and developing client applications.