Scylla Monitoring Stack¶
This document describes the setup of Scylla monitoring Stack, base on Scylla Prometheus API.
The monitoring stack needs to be installed on a dedicated server, external to the Scylla cluster. Make sure the monitoring server have access to the Scylla nodes so that it can pull the metrics over the Prometheus API.
For evaluation, you can run Scylla monitoring stack on any server (or laptop) that can handle two Docker instances at the same time. For production, see recommendation below.
Minimal Production System Recommendations¶
- CPU - at least 2 physical cores/ 4vCPUs
- Memory - 15GB+ DRAM
- Disk - 1TB+ of persistent disk storage
- Network - 1gbe/10be preferred
Scylla monitoring stack consists of two components, wrapped in Docker containers:
- prometheus - collects and stores metrics
- grafana - dashboard server
- Install scylla-grafana-monitoring
git clone https://github.com/scylladb/scylla-grafana-monitoring.git cd scylla-grafana-monitoring
- Start docker service if needed
centos $ sudo service docker start ubuntu $ sudo systemctl restart docker
prometheus/node_exporter_servers.yml with the targets IP (the server you wish to monitor).
For every server, there are two targets, one under Scylla job which is used for the scylla metrics.
Use port 9180.
targets: - 172.17.0.2:9180 - 172.17.0.3:9180
For general node information (disk, network, etc.) add the server under
prometheus/node_exporter_servers.yml job. Use port 9100.
targets: - 172.17.0.2:9100 - 172.17.0.3:9100
It is possible to configure your own target files instead of updating
node_exporter_servers.yml, using the
-s for scylla target file and
-n for node targets file.
./start-all.sh -s my_scylla_server.yml -n my_node_exporter_servers.yml -d data_dir
In many deployments the contents of those files are very similar, with the same servers being listed differing only in the ports scylla and node_exporter listen to. To automatically generate the target files, one can use the
genconfig.py script, using the
-s flags to control which files get created:
./genconfig.py -ns -d myconf 192.168.0.1 192.168.0.2
After that, the monitoring stack can be started pointing to the servers at
./start-all.sh -s myconf/scylla_server.yml -n myconf/node_exporter_servers.yml
Start and Stop¶
./start-all.sh -d data_dir
View Grafana Monitor¶
Point your browser to
By default, Grafana authentication is disabled. To enable it and set a password for user admin use the
- Choose Disk and network interface
The dashboard holds a drop down menu at its upper left corner for disk and network interface. You should choose relevant disk and interface for the dashboard to show the graphs.