How to Report a Scylla Problem

In the event there is an issue you would like to report to ScyllaDB support, you need to submit logs and other files which help the support team diagnose the issue. Only the ScyllaDB support team members can read the data you share.

The Health Check Report

The Health Check Report is a script which generates:

  • An archive file (output_files.tgz) containing configuration data (hardware, OS, Scylla SW, etc.)
  • System logs
  • A Report file (<node_IP>-health-check-report.txt) based on the collected info.

Prepare and upload the Health Check Report

Procedure:

  1. Run the node health check script: node_health_check
  2. Generate a UUID: export report_uuid=$(uuidgen)
  3. Upload the files to upload.scylladb.com. Run the following:
curl -X PUT http://upload.scylladb.com/$report_uuid/output_files.tgz -T output_files.tgz
curl -X PUT http://upload.scylladb.com/$report_uuid/[node_IP]-health-check-report.txt -T [node_IP]-health-check-report.txt

The UUID you generated replaces the variable $report_uuid at runtime.

  1. Supply ScyllaDB support with the UUID. Keep in mind that although the ID you supply is public, only ScyllaDB support team members can read the data you share. Do one of the following:
  • If you are using the Open Source version of Scylla, open an issue on GitHub and include the UUID within the issue.
  • If you are using Scylla Enterprise version, open a Support Ticket and include the UUID within the ticket.

About the node health check

There are other options avaiable when running the node health check script.

./node_health_check.sh -h

This script performs system review and generates health check report based on
the configuration data (hardware, OS, Scylla SW, etc.) collected from the node.

Usage:
-p   Port to use for nodetool commands (default: 7199)
-q   Port to use for cqlsh (default: 9042)
-c   Print cfstats output
-d   Print data model info
-n   Print network info
-a   Print all
-h   Display this help and exit

Note: output for the above is collected, but not printed in the report.
If you wish to have them printed, please supply the relevant flag/s.

Core Dump

When Scylla fails, it creates core dump which can later be used to debug the issue. The file is written to /var/lib/scylla/coredump. If there is no file in the directory, see Troubleshooting Core Dump.

Prepare and upload the core dump file

Procedure

1. The core dump file can be very large. Make sure to zip it using xz or similar. Run (for example): xz -z core.21692

  1. Upload the compressed file to upload.scylladb.com. Run:
curl -X PUT http://upload.scylladb.com/$report_uuid/yourfile -T yourfile

The UUID you generated replaces the variable $report_uuid at runtime.

  1. If you have not done so already, supply ScyllaDB support with the UUID. Keep in mind that although the ID you supply is public, only ScyllaDB support team members can read the data you share. Do one of the following:
  • If you are using the Open Source version of Scylla, open an issue on GitHub and include the UUID within the issue.
  • If you are using Scylla Enterprise version, open a Support Ticket and include the UUID within the ticket.

Troubleshooting Core Dump

In the event the core dump directory is empty, the following solutions may help.

Operating System not set to generate core dump files

If Scylla restarts for some reason and there is no core dump file, the OS system deamon needs to be modified.

Open the custom configuration file. /etc/systemd/coredump.conf.d/custom.conf. Refer to generate core dumps for details.

Note that you will need a spare disk space which is larger than Scylla RAM.

Core dump file exists, but not where you expect it to be

If the scylla/coredump directory is empty even after you changed the custom configuraiton file, it might be that Automatic Bug Reporting Tool (ABRT) is running and all core dumps are pipelined directly to it. Check the /proc/sys/kernel/core_pattern file, and if it contains something similar to |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %h %e 636f726500 replace the contents with core.

Back