Scylla University Live | Nov 9 (AMERICAS), Nov 10 (EMEA and APAC) Free Virtual Training Event
Learn More

Large Partitions Hunting

This document describes how to catch large partitions.

What Should Make You Want To Start Looking For A Large Partition?

Any of the following:

  • Latencies on a single shard become very long (look at the “Scylla Overview Metrics” Scylla Monitoring dashboard)

  • Oversized allocation warning messages in the log:

    seastar_memory - oversized allocation: 2842624 bytes, please report
  • A warning of “too many rows” is issued when writing to a table (usually happens during a compaction):

    Nov 26 07:36:29 hostname scylla[24314]:  [shard 9] large_partition - Writing a partition with too many rows [Some_KS/Some_table:PK_VAL1] (211663 rows)
    Nov 26 08:36:19 hostname scylla[24314]:  [shard 34] large_partition - Writing a partition with too many rows [Some_KS/Some_table:PK_VAL2] (171994 rows)

    In this case, refer to Troubleshooting Large Partition Tables for more information.

What To Do When You Suspect You May Have A Large Partition?

For each table you suspect run:

nodetool flush <keyspace name> <table name>
nodetool cfstats <keyspace name>.<table name> | grep "Compacted partition maximum bytes"

For example:

nodetool cfstats demodb.tmcr | grep "Compacted partition maximum bytes"

                Compacted partition maximum bytes: 1188716932

Using system tables to detect large partitions, rows, or cells

Starting from scylla 2.3, large partitions are listed in the system.large_partitions table. See Scylla Large Partitions Table for more information.

Starting from scylla 3.1, large rows and large cells are listed similarly in the system.large_rows and system.large_cells tables, respectively. See Scylla Large Rows and Cells Tables for more information.

When Compaction Creates an Error

When compaction or writing to a table results in a “Writing a partition with too many rows” warning:

This warning indicates that there is a huge multi-row partition (based on the number of rows) and it is orthogonal to the size-based warnings. The warning is controlled by compaction_rows_count_warning_threshold, which is set in the scylla.yaml. See Troubleshooting Large Partition Tables for more information.