Large Partitions Hunting

This document describes how to catch large partitions.

What Should Make You Want To Start Looking For A Large Partition?

Any of the following:

  • Latencies on a single shard become very long (look at the “Scylla Overview Metrics” Scylla Monitoring dashboard)

  • Oversized allocation warning messages in the log:

seastar_memory - oversized allocation: 2842624 bytes, please report

What To Do When You Suspect You May Have A Large Partition?

  1. For each table you suspect run:

nodetool flush <keyspace name> <table name>
nodetool cfstats <keyspace name>.<table name> | grep "Compacted partition maximum bytes"

For example:

nodetool cfstats demodb.tmcr | grep "Compacted partition maximum bytes"

                Compacted partition maximum bytes: 1188716932

Using system tables to detect large partitions, rows, or cells

Starting from scylla 2.3, large partitions are listed in the system.large_partitions table. See Scylla Large Partitions Table for more information.

Starting from scylla 3.1, large rows and large cells are listed similarly in the system.large_rows and system.large_cells tables, respectively. See Scylla Large Rows and Cells Tables for more information.