Topic: Increasing resource limits when Scylla runs and is managed via systemd
Audience: Scylla administrators
Updates to /etc/security/limits.d/scylla.conf do not have any effect. After a cluster rolling restart is completed, the Scylla limits listed under /proc/<PID>/limits are still the same or lower than what has been configured.
When running under systemd, Scylla enforces the LimitNOFILE and LimitNPROC values under /lib/systemd/system/scylla-server.service, where:
LimitNOFILE - Maximum number of file descriptors allowed to be opened simultaneously (defaults to 800000)
LimitNPROC - Maximum number of processes allowed to run in parallel (defaults to 8096)
Even though Scylla’s provided defaults are suitable for most workloads, there may be situations on which these values may need to be overriden.
The Linux kernel imposes an upper limit on the maximum number of file-handles that a process may allocate, which takes precedence over systemd. Such limit may be persistently increased by tuning the fs.nr_open parameter in the /etc/sysctl.conf file.
The fs.nr_open parameter default value is 1048576 (1024*1024) and it must be increased whenever it is required to overcome such limit.
As a rule of thumb, always ensure that the value of fs.nr_open is equal or greater than the maximum number of file-handles that Scylla may be able to consume.
To check the value of fs.nr_open run:
sysctl fs.nr_open
fs.nr_open = 1048576
Edit the file /etc/sysctl.conf as root, and increase the value of fs.nr_open:
sudo vi /etc/sysctl.conf
fs.nr_open=5000000
Save the changes and apply the new setting to the running configuration:
sudo sysctl -p
To override Scylla limits on systemd, run:
sudo systemctl edit scylla-server.service
Within the opened text editor, add the following lines and adjust the parameters as needed, e.g.:
Warning: Avoid setting the value of such fields to Infinity as this can lead to unpredictable behaviors. When adjusting the value of LimitNOFILE always set it to a value which is equal or lower than the value of fs.nr_open
[Service]
LimitNOFILE=5000000
Restart Scylla:
sudo systemctl restart scylla-server.service
This will create a configuration file named override.conf under the /etc/systemd/system/scylla-server.service.d folder. Whenever editing this file by hand manually, remember to run sudo systemctl daemon-reload before restarting Scylla, so that systemd reloads the changes.
To check the updated limits allowed by the Scylla process run:
cat /proc/$(pidof scylla)/limits
The Linux Kernel Documentation for /proc/sys/fs/* <https://www.kernel.org/doc/Documentation/sysctl/fs.txt>
systemd.exec(5) manpage