How to Report a Scylla Problem

To save time and increase the likelihoods of a solution, please follow the guidelines when reporting a Scylla problem.

Information to Include in the Report

Run the node_health_check script (Starting from Scylla version 2.0 it will be in the default path):

It will generate an archive file (output_files.tgz) containing configuration data (hardware, OS, Scylla SW, etc.) and system logs, as well as a textual report file (<node_IP>-health-check-report.txt) based on the collected info.

./node_health_check.sh -h

This script performs system review and generates health check report based on
the configuration data (hardware, OS, Scylla SW, etc.) collected from the node.

Usage:
-p   Port to use for nodetool commands (default: 7199)
-q   Port to use for cqlsh (default: 9042)
-c   Print cfstats output
-d   Print data model info
-n   Print network info
-a   Print all
-h   Display this help and exit

Note: output for the above is collected, but not printed in the report.
If you wish to have them printed, please supply the relevant flag/s.
  • Generate a UUID, export report_uuid=$(uuidgen) (this uuid will be used to upload the configuration data archive, textual report and core dump file/s)
  • Upload a file to upload.scylladb.com:
curl -X PUT http://upload.scylladb.com/$report_uuid/output_files.tgz -T output_files.tgz
curl -X PUT http://upload.scylladb.com/$report_uuid/[node_IP]-health-check-report.txt -T [node_IP]-health-check-report.txt

Core Dump

When Scylla fails, it should create a core dump which can later be used to debug the issue.

Coredumps are written to /var/lib/scylla/coredump

Systemd

If Scylla restarts for some reason and there is no core dump file, make sure you set the OS to generate core dumps. Note that you will need a spare disk space which is larger than Scylla RAM.

Core dump file location is defined in /etc/systemd/coredump.conf.d/custom.conf

No Systemd

If files are still not written, it might be that Automatic Bug Reporting Tool (ABRT) is running and all core dumps are pipelined right to it. Check /proc/sys/kernel/core_pattern file, and if it have something like |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %h %e 636f726500 replace it with just core.

Send the Core Dump

The dump file can be very large. Make sure to zip it with xz or similar: xz -z core.21692

Upload the compress file:

curl -X PUT http://upload.scylladb.com/$report_uuid/yourfile -T yourfile

Where report-uuid is the uuid you generated earlier.

Back