Scylla Monitoring Stack¶

Calculating Prometheus Minimal Disk Space requirement¶

Prometheus storage disk performance requirements: persistent block volume, for example an EC2 EBS volume

Prometheus storage disk volume requirement: proportional to the number of metrics it holds. The default retention period is 15 days, and the disk requirement is around 200MB per core, assuming the default scraping interval of 15s.

For example, when monitoring a 6 node Scylla cluster, each with 16 CPU cores, and using the default 15 days retention time, you will need minimal disk space of

6 * 16 * 200MB ~ 20GB

To account for unexpected events, like replacing or adding nodes, we recommend allocating at least x4-5 space, in this case, ~100GB. Prometheus Storage disk does not have to be as fast as Scylla disk, and EC2 EBS, for example, is fast enough and provide HA out of the box.

Prerequisites¶

• docker

Procedure¶

1. Download and extract the latest Scylla Monitoring Stack binary; for example, for release 2.2.0

wget https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-2.2.tar.gz
tar -xvf scylla-monitoring-2.2.tar.gz
cd scylla-monitoring-scylla-monitoring-2.2

As an alternative, you can clone and use the git repository directly.

git clone https://github.com/scylladb/scylla-monitoring.git
cd scylla-monitoring
git checkout branch-2.2

2. Start docker service if needed

centos $ sudo service docker start
ubuntu $ sudo systemctl restart docker

3. Update prometheus/scylla_servers.yml with the targets’ IPs (the servers you wish to monitor).

For example:

targets:
    - 172.17.0.2
    - 172.17.0.3
labels:
    cluster: cluster1
    dc: dc1

For general node information (disk, network, etc.) Scylla Monitoring Stack uses the node_exporter agent that runs on the same machine as Scylla does. By default, Prometheus will assume you have a node_exporter running on each machine. If this is not the case, you can override the node_exporter targets configuration file by creating an additional file and pass it with the -n flag.

It is possible to configure your own target file instead of updating scylla_servers.yml, using the -s for scylla target file.

For example:

./start-all.sh -s my_scylla_server.yml -d data_dir

Use Labels to mark different Data Centers

As can be seen in the examples, each target has its own set of labels to mark the cluster name and the data center (dc). You can add multiple targets in the same file for multiple clusters or multiple data centers.

4. Connect to Scylla Manager by updating prometheus/scylla_manager_servers.yml If you are using Scylla Manager, you should set its ip.

For example

# List Scylla Manager end points

- targets:
  - 127.0.0.1:56090

Note that you do not need to add labels to the Scylla Manager targets.

Start and Stop¶

Start

./start-all.sh -d data_dir

Stop

./kill-all.sh

Setting Specific Version¶

By default, start-all.sh will start with dashboards for the latest two Scylla versions and the latest Scylla Manager version.

You can specify specific scylla version with the -v flag and Scylla Manager version with -M flag

For example:

./start-all.sh -v 3.0,master -M 1.3

will load the dashboards for Scylla versions 3.0 and master and the dashboard for Scylla Manager 1.3

View Grafana Monitor¶

Point your browser to your-server-ip:3000 By default, Grafana authentication is disabled. To enable it and set a password for user admin use the -a option

For example:

10.10.0.1:3000

• Choose Disk and network interface

The dashboard holds a drop down menu at its upper left corner for disk and network interface. You should choose relevant disk and interface for the dashboard to show the graphs.