System Configuration Guide

System configuration steps are performed automatically by the Scylla RPM and deb packages. For information on getting started with Scylla, see Getting Started.

All Scylla AMIs and Docker images are pre-configured with the following steps, by a script. This document is provided for reference.

Hardware recommendations

  • Storage: For maximum performance SSD devices are highly recommended. When using more than one, use RAID0 to couple them together for the Scylla data directory.
  • Networking: 10Gbit/s cards.
  • Scylla is CPU intensive. Do not run additional CPU intensive tasks on the same server/cores as Scylla.

System configuration files and scripts

Several system configuration settings should be applied. For ease of use, the necessary scripts and configuration files are provided. Files are under dist/common and seastar/scripts in the Scylla source code, and installed in the appropriate system locations. (For information on Scylla’s own configuration file, see Scylla Configuration.)

System configuration files

Source file Installed location Description
limits.d/scylla.conf /etc/security/limits.d/scylla.conf Remove system resource limits
sysconfig/scylla-server /etc/sysconfig/scylla-server Server startup options
(written by scripts/scylla_coredump_setup , below) /etc/sysconfig/sysctl.d/99-scylla.conf Configure core dumps to use the scylla_save_coredump script

Setup scripts

The following scripts are documented for reference purposes. All of them are invoked by the scylla_setup script, which should be run at install time, or when the system hardware changes.

Source file Installed location Description
scripts/scylla_bootparam_setup /usr/lib/scylla/scylla_bootparam_setup Set kernel options in bootloader
scripts/scylla_coredump_setup /usr/lib/scylla/scylla_coredump_setup Remove crash reporting software and set pattern for core dump names
scripts/scylla_ntp_setup /usr/lib/scylla/scylla_ntp_setup Configure Network Time Protocol
scripts/scylla_prepare /usr/lib/scylla/scylla_prepare Set up RAID and invoke network configuration
scripts/scylla_raid_setup /usr/lib/scylla/scylla_raid_setup Configure RAID and make XFS filesystem
scripts/scylla_run /usr/lib/scylla/scylla_run Wrapper to run Scylla with arguments from environment
scripts/scylla_save_coredump /usr/lib/scylla/save_coredump Compress a core dump file (Ubuntu only)
scripts/scylla_stop /usr/lib/scylla/scylla_stop Reset network mode if running in virtio or DPDK mode
scripts/scylla_sysconfig_setup /usr/lib/scylla/sysconfig_setup Rewrite the /etc/sysconfig/scylla file
seastar/scripts/posix_net_conf.sh /usr/lib/scylla/posix_net_conf.sh Set up networking options

System limits

The following system limits should be set or removed.

Name Required value Description
core unlimited maximum core dump size
memlock unlimited maximum locked-in-memory address space
nofile 200000 maximum number of open file descriptors
as unlimited address space limit
nproc 8096 maximum number of processes

This configuration is provided in the file /etc/security/limits.d/scylla.conf.

Bootloader settings

If Scylla is installed on an Amazon AMI, the bootloader should provide the clocksource=tsc and tsc=reliable options. This enables an accurate, high-resolution Time Stamp Counter (TSC) for setting the system time.

This configuration is provided in the file /usr/lib/scylla/scylla_bootparam_setup.

Remove crash reporting software

Remove the apport-noui or abrt packages if present, and set up a location and file name pattern for core dumps.

This configuration is provided in the file /usr/lib/scylla/scylla_bootparam_setup.

Set up network time synchronization

It is highly recommended to enforce time synchronization between Scylla servers.

Run ntpstat on all nodes to check that system time is synchronized. If you are running in a virtualized environment and your system time is set on the host, you may not need to run NTP on the guest. Check the documentation for your platform.

If you have your own time servers shared with an application using Scylla, use the same NTP configuration as for your application servers. The script /usr/lib/scylla/scylla_ntp_setup provides sensible defaults, using Amazon NTP servers if installed on the Amazon cloud, and other pool NTP servers otherwise.

Set up RAID and Filesystem

Setting the file system to XFS is the most important and mandatory for production. Scylla will significantly slow down without it.

The script /usr/lib/scylla/scylla_raid_setup performs the necessary RAID configuration and XFS filesystem creation for Scylla.

Arguments to the script are

  • -d specify disks for RAID
  • -r MD device name for RAID
  • -u update /etc/fstab for RAID

On the Scylla AMI, the RAID configuration is handled automatically in the /usr/lib/scylla/scylla_prepare script.

Networking

On AWS:

  1. Prevent irqbalance from moving your NICs’ IRQs.
  2. Bind all NICs’ HW queues to CPU0:
for irq in `cat /proc/interrupts | grep <networking iface name> | cut -d":" -f1`
do echo "Binding IRQ $irq to CPU0" echo 1 > /proc/irq/$irq/smp_affinity done
  1. Enable RPS and bind RPS queues to CPUs other than CPU0 and its hyper-threading siblings.
  2. Enable XPS and distribute all XPS queues among all available CPUs.

The posix_net_conf.sh script does all of the above.*

On bare metal setups with multi-queue NICs

  1. Prevent irqbalance from moving your NICs IRQs.
  2. Bind each NIC’s IRQ to a separate CPU.
  3. Enable XPS exactly the same way as for AWS above.
  4. Set higher values for a listen() socket backlog and for unacknowledged pending connections backlog:
echo 4096 > /proc/sys/net/core/somaxconn
echo 4096 > /proc/sys/net/ipv4/tcp_max_syn_backlog

The posix_net_conf.sh script with the -mq parameter does all of the above.

Configuring Scylla

Configuration for Scylla itself is in the Scylla Configuration section of the administration guide.

Development system configuration

The following item is not required in production.

When working on DPDK support for Scylla, enable hugepages.

NR_HUGEPAGES=128
mount -t hugetlbfs -o pagesize=2097152 none /mnt/huge
mount -t hugetlbfs -o pagesize=2097152 none /dev/hugepages/
for n in /sys/devices/system/node/node?; do
    echo $NR_HUGEPAGES > $n/hugepages/hugepages-2048kB/nr_hugepages;
done

Huge page configuration is written to /etc/sysconfig/scylla-server by the script /usr/lib/scylla/sysconfig_setup.