Large Partitions Hunting¶
This document describes how to catch large partitions.
Any of the following:
Latencies on a single shard become very long (look at the “Scylla Overview Metrics” Scylla Monitoring dashboard)
Oversized allocation warning messages in the log:
seastar_memory - oversized allocation: 2842624 bytes, please report
A warning of “too many rows” is issued when writing to a table (usually happens during a compaction):
Nov 26 07:36:29 hostname scylla: [shard 9] large_partition - Writing a partition with too many rows [Some_KS/Some_table:PK_VAL1] (211663 rows) Nov 26 08:36:19 hostname scylla: [shard 34] large_partition - Writing a partition with too many rows [Some_KS/Some_table:PK_VAL2] (171994 rows)
In this case, refer to Troubleshooting Large Partition Tables for more information.
For each table you suspect run:
nodetool flush <keyspace name> <table name> nodetool cfstats <keyspace name>.<table name> | grep "Compacted partition maximum bytes"
nodetool cfstats demodb.tmcr | grep "Compacted partition maximum bytes" Compacted partition maximum bytes: 1188716932
Starting from scylla 2.3, large partitions are listed in the
system.large_partitions table. See Scylla Large Partitions Table for more information.
Starting from scylla 3.1, large rows and large cells are listed similarly in the
system.large_cells tables, respectively. See Scylla Large Rows and Cells Tables for more information.
When compaction or writing to a table results in a “Writing a partition with too many rows” warning:
This warning indicates that there is a huge multi-row partition (based on the number of rows) and it is orthogonal
to the size-based warnings. The warning is controlled by
compaction_rows_count_warning_threshold, which is set in the scylla.yaml.
See Troubleshooting Large Partition Tables for more information.