Troubleshoot Monitoring¶

Solution¶

If there are no data points, or if a node appears to be unreachable when you know it is, the immediate suspect is the Prometheus connectivity.

Start by login to the Prometheus console:

point your browser to http://{ip}:9090 ({ip} is the Prometheus ip address).

Go to the target tabs: http://{ip}:9090/targets and see if any of the targets is down and if there is an error message.

• Not using the local network for local ip range

When using Docker containers, by default, the local ip range (127.0.0.X) is inside the Docker container and not the host local address, if you are trying to connect to a target via the local ip range from inside a Docker container, you need to use the -l flag to enable local network stack.

• Prometheus may be pointing to the wrong target. Check your prometheus/scylla_servers.yml and prometheus/node_exporter_servers.yml. Make sure in both cases Prometheus is pulling data from the Scylla server.

Or

• Your dashboard and Scylla version may not be aligned. If you are running Scylla 2.0.x, you need to start the monitoring server with ./start-all.sh.

For example:

./start-all.sh -v 2.0.1

More on start-all.sh options.

Solution¶

1. Check Prometheus is running using sudo docker ps. If it is not running check the prometheus.yml for errors.

For example:

CONTAINER ID  IMAGE    COMMAND                  CREATED         STATUS         PORTS                                                    NAMES
41bd3db26240  monitor  "/docker-entrypoin..."   25 seconds ago  Up 23 seconds  7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp   monitor

2. If it is running, go to “Data Source” in the Grafana GUI, choose Prometheus and click Test Connection.

Solution¶

• use curl <scylla_node>:9180/metrics to fetch binary metric data from Scylla. If curl does not return data, the problem is the connectivity between the monitoring and Scylla server. Please check your IPs and firewalls.

For example

curl 172.17.0.2:9180/metrics

Solution¶

1. Make sure node_exporter is running on each Scylla server. node_exporter is installed by scylla_setup. If it does not, make sure to install and run it.

2. If it is running, use curl <scylla_node>:9100/metrics (where 172.17.0.2 is a Scylla server IP) to fetch binary metric data from Scylla. If curl does not return data, the problem is the connectivity between the monitoring and Scylla server. Please check your IPs and firewalls.

Notice to users upgrading to Scylla Open Source 3.0 or Scylla Enterprise 2019.1¶

While upgrading you need to upgrade the node_exporter from 0.14 to 0.17 version.

If the node_exporter service is not starting it may be that it needs to be updated manually.

Check the node_exporter version node_exporter --version if it shows 0.14 check the node_exporter section in the upgrade guide.