Extract Cluster Metrics in Prometheus format

Scylla clusters collect and send metrics regarding their workloads and health. These metrics, by default, are collected and sent to the Scylla Cloud Prometheus server and are monitored by the Scylla Cloud team. The most important subset of these metrics is available on the Scylla Cloud monitoring page in Scylla Cloud web UI.

Customers who wish to access these metrics outside of the Scylla Cloud service and use them with their own monitoring tool can now use the Prometheus Proxy to do so. You can use the Prometheus Proxy to extract data to an external 3rd party tool that reads Prometheus format such as DataDog and others.

This guide explains how to extract your cluster metrics and monitor them via the Scylla Monitoring tool (which includes the Prometheus server in its stack). If you wish to use the metrics otherwise, you are required to install the Prometheus server, or to use 3rd party tool that can read Prometheus output files. Instructions on transferring the metrics to Datadog are available in the following blog.

Before you begin

The following documents should be consulted:

Extract ScyllaDB Cluster metrics

Procedure

  1. Install the latest version of Scylla Monitor.

  2. Start Scylla Monitor and validate it was installed successfully.

  3. Open a support ticket and request to export the cluster metrics. Make sure to mention your cluster_id number in your request. This ID can be retrieved from Scylla Cloud from the Monitor page on the top left side of the page.

  4. As a response to your request, you will receive a Prometheus Configuration file. For example:

    scrape_configs:
- job_name: scylla_cloud_cluster_2
  scrape_interval: 15s
  honor_labels: true
  metrics_path: /api/v1/cluster/2/proxy/federate
  params:
    match[]:
    - '{job=~".+"}'
  bearer_token: SECRET_TOKEN
  static_configs:
  - targets:
    - api.cloud.example.com

  5. Navigate to the scylla-monitoring repository:

    cd scylla-monitoring

  6. Create a scylla-cloud.yaml file and add the Prometheus Configuration file you received to it.

  7. Edit the prometheus/prometheus.yml.template manually and add additional configuration settings from the scrape_configs of scylla-cloud.yaml file you just saved.

    Alternatively, using the yq tool, merge two files using the following command:

    yq merge -i -a prometheus/prometheus.yml.template ../scylla-cloud.yaml

  8. Restart scylla-monitoring service.

  9. Point your browser to the URL indicated to view your cluster metrics in Grafana.