Scylla Monitoring Stack

This document describes the setup of Scylla monitoring Stack, base on Scylla Prometheus API.

The monitoring stack needs to be installed on a dedicated server, external to the Scylla cluster. Make sure the monitoring server have access to the Scylla nodes so that it can pull the metrics over the Prometheus API.

For evaluation, you can run Scylla monitoring stack on any server (or laptop) that can handle two Docker instances at the same time. For production, see recommendation below.

Minimal Production System Recommendations

  • CPU - at least 2 physical cores/ 4vCPUs
  • Memory - 15GB+ DRAM
  • Disk - 1TB+ of persistent disk storage
  • Network - 1gbe/10be preferred

Scylla monitoring stack consists of two components, wrapped in Docker containers:

  • prometheus - collects and stores metrics
  • grafana - dashboard server
../../../_images/monitor.png

Prerequisites

  • git
  • docker

Procedure

  1. Install scylla-grafana-monitoring
git clone https://github.com/scylladb/scylla-grafana-monitoring.git
cd scylla-grafana-monitoring
  1. Start docker service if needed
centos $ sudo service docker start
ubuntu $ sudo systemctl restart docker

3. Update prometheus/scylla_servers.yml and prometheus/node_exporter_servers.yml with the targets IP (the server you wish to monitor). For every server, there are two targets, one under Scylla job which is used for the scylla metrics. Use port 9180.

For example:

targets:
- 172.17.0.2:9180
- 172.17.0.3:9180

For general node information (disk, network, etc.) add the server under prometheus/node_exporter_servers.yml job. Use port 9100. For example:

targets:
- 172.17.0.2:9100
- 172.17.0.3:9100

It is possible to configure your own target files instead of updating scylla_servers.yml and node_exporter_servers.yml, using the -s for scylla target file and -n for node targets file.

For example:

./start-all.sh -s my_scylla_server.yml -n my_node_exporter_servers.yml -d data_dir

In many deployments the contents of those files are very similar, with the same servers being listed differing only in the ports scylla and node_exporter listen to. To automatically generate the target files, one can use the genconfig.py script, using the -n and -s flags to control which files get created:

./genconfig.py -ns -d myconf 192.168.0.1 192.168.0.2

After that, the monitoring stack can be started pointing to the servers at 192.168.0.1 and 192.168.0.2 with:

./start-all.sh -s myconf/scylla_server.yml -n myconf/node_exporter_servers.yml

Start and Stop

Start

./start-all.sh -d data_dir

Stop

./kill-all.sh

View Grafana Monitor

Point your browser to your-server-ip:3000 By default, Grafana authentication is disabled. To enable it and set a password for user admin use the -a option

For example:

10.10.0.1:3000
  • Choose Disk and network interface

The dashboard holds a drop down menu at its upper left corner for disk and network interface. You should choose relevant disk and interface for the dashboard to show the graphs.