Scylla will not Start

Problem

The scylla process stopped hours ago and it won’t start

How to Verify

Possible cause: The Scylla process is managed by systemd, and systemd expects it to be able to fully start within a timeout. If this timeout is reached systemd it will kill the Scylla process and try to start it again. If that is the case, you will see the following message in the Scylla logs:

systemd[1]: scylla-server.service start operation timed out. Terminating.

The timeout in systemd should be enough to always start the server, but some versions of Scylla prior to 3.0 suffered from an issue in which the startup process could be slowed down. Even in the absence of issues since the timeout is just a number, there may be extreme cases in which it is not enough.

Solution

  1. Locate the directory with the systemd files where the scylla-server.service resides.

For Centos operating systems it is expected to be under /usr/lib/systemd/system/scylla-server.service

For Ubuntu operating systems it is expected to be under /etc/systemd/system/scylla-server.service.d

  1. Create the following directory (if not exist)

Centos

sudo mkdir /usr/lib/systemd/system/scylla-server.service

Ubuntu

sodu mkdir /etc/systemd/system/scylla-server.service.d
  1. Create a file inside that directory named 10-timeout.conf, with the following contents:
[Service]
TimeoutStartSec=9000
  1. Reload the systemd Daemon for the new configurations to take in effect.
systemctl daemon-reload