Deploying Scylla Monitoring Without Docker

The following instructions will help to deploy Scylla Monitoring Stack in the case you can not use the docker version.

Please note, Scylla recommends you use the docker version as it will provide you with most updated, current scylla monitoring system.

Minimal Production System Recommendations

  • CPU - at least 2 physical cores/ 4vCPUs
  • Memory - 15GB+ DRAM
  • Disk - 1TB+ of persistent disk storage
  • Network - 1gbe/10be preferred

The main item to set an alert on is the available disk space in the monitoring system. Data is indefinitely accrued on the Prometheus data directory. The current monitoring solution does not churn data.

Install Scylla Monitor

The following procedure is using a CentOS 7 based instance

  1. Install git

sudo yum install git -y

  1. Clone the Scylla monitoring solution

sudo git clone https://github.com/scylladb/scylla-grafana-monitoring.git

Install Prometheus

Tested with prometheus 1.7.1 version

  1. Install Prometheus
  1. Copy the following files: prometheus.yml, scylla_servers.yml and node_exporter_servers.yml from scylla-grafana-monitoring/prometheus directory to prometheus installation directory.

For example:

cp -p /home/centos/scylla-grafana-monitoring/prometheus/*.yml /home/centos/prometheus-1.7.1.linux-amd64
  1. Edit the prometheus.yml file to point to the correct static data sources.

Note

Make sure to include the honor_labels: true parameter in the prometheus.yml file.

sudo vi /<prometheus_installation_location>/prometheus-1.7.1.linux-amd64

For example:

global:
  scrape_interval: 5s # By default, scrape targets every 5 second.
  scrape_timeout: 4s # Timeout before trying to scape a target again

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'scylla-monitor'

scrape_configs:
- job_name: scylla
  honor_labels: true
  file_sd_configs:
    - files:
      - /home/centos/prometheus-1.7.1.linux-amd64/scylla_servers.yml

- job_name: node_exporter
  honor_labels: true
  file_sd_configs:
    - files:
      - /home/centos/prometheus-1.7.1.linux-amd64/node_exporter_servers.yml
  1. Edit the, scylla_servers.yml and node_exporter_servers.yml files to point to your Scylla nodes.

scylla_servers.yml:

For example:

cat scylla_servers.yml
# List Scylla end points

- targets:
  - 192.168.66.6:9180
  - 192.168.66.244:9180

node_exporter_servers.yml:

For example:

cat node_exporter_servers.yml
# List end points with node exporter

- targets:
  - '192.168.66.6:9100'
  - '192.168.66.244:9100'
  1. Create a data directory for prometheus to store the metrics data

For example:

sudo mkdir /home/centos/prometheus-1.7.1.linux-amd64/mydata
  1. Start Prometheus server:

For example:

sudo ./prometheus --config.file=prometheus.yml --storage.tsdb.path /home/centos/prometheus-1.7.1.linux-amd64/mydata

Data should start accumulate on: /home/centos/prometheus-1.7.1.linux-amd64/mydata

  1. Verify that Prometheus is up and running, point your browser to the Prometheus IP:Port

For example:

http://192.168.135.166:9090/
../../../_images/1.png

Prometheus console should be visible

  1. Verify that the collectd and scylla metrics accumulating in the server by executing a query through the console

For example:

node_memory_MemFree

../../../_images/2.png

And

scylla_reactor_utilization

../../../_images/3.png

At this point Scylla is emitting the metrics and Prometheus is able to store them.

Install Grafana

  1. Install Grafana base on the instructions here
  1. Start the Grafana service

sudo service grafana-server start

  1. Access to the cloned Scylla Grafana directory

cd /home/centos/scylla-grafana-monitoring/

  1. Start the Grafana dashboards

For example:

sudo ./load-grafana.sh -p 192.168.135.166:9090 -g 3000 -a admin -v 2.0
  • The IP address above is the prometheus server IP address (192.168.135.166) and the port it uses (9090).
  • The -g 3000 is defining the Grafana localhost port
  • The -a admin is the default password for the Grafana admin users, feel free to change it :)
  • The -v is optional Scylla version selection, if omitted, Grafana will upload all available dashboards.

Point your browser to the Grafana server port 3000, the assumption is that Grafana and Prometheus are collocated on the same server.

../../../_images/4.png

Monitoring Scylla