Topic: Increasing resource limits when Scylla runs and is managed via systemd
Audience: Scylla administrators
/etc/security/limits.d/scylla.conf do not have any effect. After a cluster rolling restart is completed, the Scylla limits listed under
/proc/<PID>/limits are still the same or lower than what has been configured.
When running under systemd, Scylla enforces the LimitNOFILE and LimitNPROC values under
LimitNOFILE - Maximum number of file descriptors allowed to be opened simultaneously (defaults to 800000)
LimitNPROC - Maximum number of processes allowed to run in parallel (defaults to 8096)
Even though Scylla’s provided defaults are suitable for most workloads, there may be situations on which these values may need to be overriden.
The Linux kernel imposes an upper limit on the maximum number of file-handles that a process may allocate, which takes precedence over systemd. Such limit may be persistently increased by tuning the
fs.nr_open parameter in the
fs.nr_open parameter default value is 1048576 (1024*1024) and it must be increased whenever it is required to overcome such limit.
As a rule of thumb, always ensure that the value of
fs.nr_open is equal or greater than the maximum number of file-handles that Scylla may be able to consume.
To check the value of
sysctl fs.nr_open fs.nr_open = 1048576
Edit the file
/etc/sysctl.conf as root, and increase the value of
sudo vi /etc/sysctl.conf fs.nr_open=5000000
Save the changes and apply the new setting to the running configuration:
sudo sysctl -p
To override Scylla limits on systemd, run:
sudo systemctl edit scylla-server.service
Within the opened text editor, add the following lines and adjust the parameters as needed, e.g.:
Warning: Avoid setting the value of such fields to Infinity as this can lead to unpredictable behaviors. When adjusting the value of LimitNOFILE always set it to a value which is equal or lower than the value of
[Service] LimitNOFILE=5000000 LimitNPROC=32384
sudo systemctl restart scylla-server.service
This will create a configuration file named
override.conf under the
/etc/systemd/system/scylla-server.service.d folder. Whenever editing this file by hand manually, remember to run
sudo systemctl daemon-reload before restarting Scylla, so that systemd reloads the changes.
To check the updated limits allowed by the Scylla process run:
cat /proc/$(pidof scylla)/limits
The Linux Kernel Documentation for /proc/sys/fs/* <https://www.kernel.org/doc/Documentation/sysctl/fs.txt>