Scylla Monitoring Stack¶
This document describes the setup of Scylla monitoring Stack, base on Scylla Prometheus API.
Scylla monitoring stack consists of three components, wrapped in Docker containers:
- prometheus - collects and stores metrics
- alertmanager - handles alerts
- grafana - dashboard server
The monitoring stack needs to be installed on a dedicated server, external to the Scylla cluster. Make sure the monitoring server have access to the Scylla nodes so that it can pull the metrics over the Prometheus API.
For evaluation, you can run Scylla monitoring stack on any server (or laptop) that can handle two Docker instances at the same time. For production, see recommendation below.
Minimal Production System Recommendations¶
- CPU - at least 2 physical cores/ 4vCPUs
- Memory - 15GB+ DRAM
- Disk - persistent disk storage is porportional to the number of cores and Prometheus retention period (see the following section)
- Network - 1GbE/10GbE preferred
Calculating Prometheus Minimal Disk Space requirement¶
Prometheus storage disk performance requirements: persistent block volume, for example an EC2 EBS volume
Prometheus storage disk volume requirement: proportional to the number of metrics it holds. The default retention period is 15 days, and the disk requirement is around 200MB per core, assuming the default scraping interval of 15s.
For example, when monitoring a 6 node Scylla cluster, each with 16 CPU cores, and using the default 15 days retention time, you will need minimal disk space of
6 * 16 * 200MB ~ 20GB
To account for unexpected events, like replacing or adding nodes, we recommend allocating at least x4-5 space, in this case, ~100GB. Prometheus Storage disk does not have to be as fast as Scylla disk, and EC2 EBS, for example, is fast enough and provide HA out of the box.
Docker Post Installation¶
Docker post installation guide can be found here
Avoid running container as root.
To avoid running docker as root, you should add the user you are going to use to start the monitoring to the docker group.
- Create the docker group.
sudo groupadd docker
- Add your user to the docker group.
sudo usermod -aG docker $USER
- Start docker by calling:
sudo systemctl enable docker
- Download and extract the latest Scylla Monitoring Stack binary; for example, for release 3.0.
wget https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-2.4.tar.gz tar -xvf scylla-monitoring-2.4.tar.gz cd scylla-monitoring-scylla-monitoring-2.4
As an alternative, you can clone and use the git repository directly.
git clone https://github.com/scylladb/scylla-monitoring.git cd scylla-monitoring git checkout branch-2.4
- Start docker service if needed
centos $ sudo service docker start ubuntu $ sudo systemctl restart docker
prometheus/scylla_servers.ymlwith the targets’ IPs (the servers you wish to monitor).
It is important that the dc in the target files will match the datacenters names used by Scylla.
nodetool status command to validate the datacenter names used by Scylla.
targets: - 172.17.0.2 - 172.17.0.3 labels: cluster: cluster1 dc: dc1
If you want to add your managed cluster to Scylla Monitoring, add the IPs of the nodes as well as the cluster name you used when you added the cluster to Scylla Manager. It is important that the label
cluster name and the cluster name in Scylla Manager match.
For general node information (disk, network, etc.) Scylla Monitoring Stack uses the node_exporter agent that runs on the same machine as Scylla does.
By default, Prometheus will assume you have a node_exporter running on each machine. If this is not the case, you can override the node_exporter
targets configuration file by creating an additional
file and pass it with the
By default, there is no need to configure
node_exporter_server.yml. Prometheus will use the same targets it uses for
Sylla and will assume you have a node_exporter
running on each Scylla server.
It is possible to configure your own target file instead of updating
scylla_servers.yml, using the
-s for scylla target file.
./start-all.sh -s my_scylla_server.yml -d data_dir
Use Labels to mark different Data Centers
As can be seen in the examples, each target has its own set of labels to mark the cluster name and the data center (dc). You can add multiple targets in the same file for multiple clusters or multiple data centers.
You can use the
genconfig.py script to generate the server file.
./genconfig.py -d myconf -dc dc1:192.168.0.1,192.168.0.2 -dc dc2:192.168.0.3,192.168.0.4
This will generate a server file for four servers in two datacenters server
192.168.0.2 in dc1 and
192.168.0.4 in dc2.
After that, the monitoring stack can be started with:
./start-all.sh -s myconf/scylla_server.yml
genconfig.py script can use
nodetool status to generate the server file using the
nodetool status | ./genconfig.py -NS
4. Connect to Scylla Manager by updating
If you are using Scylla Manager, you should set its ip.
# List Scylla Manager end points - targets: - 127.0.0.1:56090
Note that you do not need to add labels to the Scylla Manager targets.
Start and Stop¶
./start-all.sh -d data_dir
Setting Specific Version¶
By default, start-all.sh will start with dashboards for the latest two Scylla versions and the latest Scylla Manager version.
You can specify specific scylla version with the
-v flag and Scylla Manager version with
./start-all.sh -v 3.0,master -M 1.3
will load the dashboards for Scylla versions
master and the dashboard for Scylla Manager
View Grafana Monitor¶
Point your browser to
By default, Grafana authentication is disabled. To enable it and set a password for user admin use the