sctool Reference

sctool is a CLI for the Scylla Manager server. The server provides a REST API used by sctool. The daemon communicates with multiple Scylla clusters and performs cluster-wide operations such as automatic repair.

Usage:

sctool command [flags] [global flags]

Global Flags:

Use sctool cluster [command] --help for more information about a command.

Managing Cluster

sctool cluster - Add or delete clusters

sctool cluster command [flags]

Available Commands:

  • add - Adds a cluster to manager
  • delete - Deletes a cluster from manager
  • list - Shows managed clusters
  • update - Modifies a cluster

Flags: -h, --help - Help for cluster command

Global Flags: see above

Add - Adding a new cluster to manage

sctool cluster add --hosts hosts-list --name cluster-name --shard-count count

A Scylla cluster needs to be registered before management tasks can be initiated.

  • hosts-list : a mandatory flag requiring a comma-separated list of a subset of Scylla nodes, which will be used to learn the cluster topology. An initial connection to each host will be attempted to validate all hosts belong to the same Scylla cluster and reside in the same datacenter. Once a Scylla cluster is registered, a Repair Unit is automatically created for each existing keyspace in the cluster, including system keyspaces. Additionally, a repair_auto_schedule task is added to be run every week.
  • shard-count : specifies the number of Scylla shards on each node in the managed Scylla cluster, it is a mandatory parameter and should match the actual number as closely as possible.
  • name: When a Scylla cluster is added, it is assigned a unique identifier which is printed to the command line. It is possible to use the --name flag to set a more user-friendly name so that it can be used instead for identifying the cluster with all the commands.

Usage example:

sctool cluster add --hosts 192.168.0.1,192.168.0.2 -n scylla-cluster-a1 --shard-count 16

Repair

sctool repair - Schedule, manage and monitor recurrent repairs

sctool repair [command] [options]

Available Commands:

  • progress Shows repair progress
  • schedule Schedule repair of a unit
  • unit Manage repair units

Flags: -h, --help - Help for repair command

Global Flags: see above

Managing repair units

sctool repair unit [command]

Available Commands:

  • add - Creates a new repair unit
  • delete - Deletes a repair unit
  • list - Shows available repair units
  • update - Modifies a repair unit

Adding a repair unit

sctool repair unit add -k keyspace [-n name] [-t table-list] [flags]

  • keyspace (required) - Keyspace to repair
  • name - A more user friendly name so that it can be used to identify the repair unit with all related commands.
  • table-list - Comma-separated list of tables in to repair in the keyspace, if empty repair the whole keyspace

Flags: -h, --help - Help for repair unit command

Global Flags: see above

Before a repair can be scheduled on a managed Scylla cluster, a Repair Unit needs to be created. Note that once a Scylla cluster is registered, a Repair Unit is automatically created for each existing keyspace in the cluster, including system keyspaces. These units are kept in sync by the automatic repair. Therefore this step is only necessary when keyspaces are added/removed before the auto repair sync had a chance to run and a repair needs to be scheduled on them. The mandatory flag --keyspace specifies the keyspace to be repaired. Specific tables in the keyspace can be selected for repair with the --tables flag. Without it (or if set to an empty list), all tables in the keyspace are to be repaired. When a repair unit is created, it is assigned a unique identifier which is printed to the command line. It is possible to use the --name flag to set a more user friendly name so that it can be used instead for identifying it with further commands.

Scheduling a repair

sctool repair schedule <unit-id> [--interval days] [--num-retries retries] [--start-date date-string] [flags]

  • days - Task schedule interval in days (default 7)
  • retries - Task schedule number of retries (default 3)
  • date-string - Task start date in RFC3339 form or now[+duration] (default “now”)

Flags: -h, --help- Help for repair schedule command

Global Flags: see above

Requesting repair of a unit is done with the sctool repair schedule command, and specifying the unique identifier or name (if one was assigned with the --name flag) of the repair unit. By default a repair task will be created starting “now”, repeating every 7 days with 3 retry attempts. The --start-date flag can be used to schedule a repair in the future by either specifying an RFC3339 format date (i.e. 2017-12-30T23:00:00Z) or the string “now” with an optional modifier specifying time units to add. For example now+2h to start 2 hours from now, now+30m to start in 30 minutes. A repair task that failed will be attempted again every 10 minutes for the number of times specified with the --retries flag in addition to the first attempt (set the value to 0 to disable retries). When a repair is scheduled, a repair task is created, it is assigned a unique identifier with the “repair” prefix (“repair/UUID”) which is printed to the command line.

Monitoring repair progress

sctool repair progress [repair/task-id] [--details] [-u unit] [flags]

  • repair/task-id - The repair task id, as returned by the sctool repair schedule command
  • unit - The repair unit name as set in the sctool repair unit add command

Flags: --details - More detailed progress data -h, --help- Help for repai progress command

Global Flags: see above

The details of the latest run (or still running) repair task can be displayed with the command sctool repair progress repair/task-id. Alternatively, without specifying the repair/task-id one can use the -u flag to get the details of the latest repair run on the specified unit. The output returned includes the repair status, overall repair progress percentage and a table listing per-host progress. Without the –details flag specified each row holds the host’s address, progress percentage and the number of failed segments. A host on which repair on its primary ranges has not started yet, will hold minus (“-”) signs instead of values

Example:

sctool repair progress repair/b509ac5d-c10a-48e7-a9f1-a938cb2d1db7
Status: running, progress: 45%
╭─────────────┬──────────┬─────────────────╮
│ host        │ progress │ failed segments │
├─────────────┼──────────┼─────────────────┤
│ 172.16.1.2  │ -        │ -               │
│ 172.16.1.3  │ 100      │ 0               │
│ 172.16.1.10 │ 37       │ 0               │
╰─────────────┴──────────┴─────────────────╯

The --details flag shows each host’s shard repair progress, with the shards numbered from zero.

Example:

sctool repair progress repair/b509ac5d-c10a-48e7-a9f1-a938cb2d1db7 --details
Status: running, progress: 44%
╭─────────────┬───────┬──────────┬─────────────────╮
│ host        │ shard │ progress │ failed segments │
├─────────────┼───────┼──────────┼─────────────────┤
│ 172.16.1.2  │ -     │ -        │ -               │
│ 172.16.1.3  │ 0     │ 100      │ 0               │
│ 172.16.1.3  │ 1     │ 100      │ 0               │
│ 172.16.1.10 │ 0     │ 34       │ 0               │
│ 172.16.1.10 │ 1     │ 33       │ 0               │
╰─────────────┴───────┴──────────┴─────────────────╯

Managing tasks

sctool task - Manage tasks

sctool task command [flags]

Available Commands:

  • delete - Deletes a task schedule
  • history - List run history of a task
  • list - Shows available tasks and their last run status
  • start - Starts executing a task
  • stop - Stops executing a task
  • update - Modifies a task

Flags: -h, --help - Help for the task command

Global Flags: see above

Listing scheduled tasks

The sctool task list command displays a table with all scheduled task for the specified cluster. Each row will hold the task id and its type separated by a slash (“/”), the original task schedule. Comprised of a start date, interval days and number of retries. Additionally the last run details are displayed which are comprised of the run start & stop dates and the last run status. Missing last run fields are filled with a minus (“-”) if the task has not been run for example, or for a running task the run stop time has not been set yet. Example:

sctool task list
╭───────────────────────────────────────────────────────────┬──────┬──────────────────────┬───────────────┬─────────────┬──────────────────────┬─────────────────────────┬─────────╮
│ task id                                                   │ name │ start date           │ interval days │ num retries │ run start            │ run stop                │ status  │
├───────────────────────────────────────────────────────────┼──────┼──────────────────────┼───────────────┼─────────────┼──────────────────────┼─────────────────────────┼─────────┤
│ repair/b509ac5d-c10a-48e7-a9f1-a938cb2d1db7               │      │ 2017-12-20T15:23:40Z │ 0             │ 0           │ 2017-12-20T15:23:40Z │ 2017-12-20T15:25:12.08Z │ stopped │
│ repair_auto_schedule/bd5e6178-ccfa-4611-a000-4d6a862857d0 │      │ 2017-12-21T00:00:00Z │ 7             │ 6           │ -                    │ -                       │ -       │
╰───────────────────────────────────────────────────────────┴──────┴──────────────────────┴───────────────┴─────────────┴──────────────────────┴─────────────────────────┴─────────╯

Setting the --all flag will also list disabled tasks which are not shown in the regular view by displaying an additional column “enabled”. Example:

╭─────────┬───────────────────────────────────────────────────────────┬──────┬──────────────────────┬───────────────┬─────────────┬──────────────────────────┬──────────────────────────┬─────────╮
│ enabled │ task id                                                   │ name │ start date           │ interval days │ num retries │ run start                │ run stop                 │ status  │
├─────────┼───────────────────────────────────────────────────────────┼──────┼──────────────────────┼───────────────┼─────────────┼──────────────────────────┼──────────────────────────┼─────────┤
│ ✓       │ repair/b509ac5d-c10a-48e7-a9f1-a938cb2d1db7               │      │ 2017-12-20T15:23:40Z │ 0             │ 0           │ 2017-12-20T16:20:17.712Z │ 2017-12-20T16:21:03.85Z  │ stopped │
│         │ repair_auto_schedule/bd5e6178-ccfa-4611-a000-4d6a862857d0 │      │ 2017-12-21T00:00:00Z │ 7             │ 6           │ 2017-12-20T16:55:28.288Z │ 2017-12-20T16:55:30.007Z │ stopped │
╰─────────┴───────────────────────────────────────────────────────────┴──────┴──────────────────────┴───────────────┴─────────────┴──────────────────────────┴──────────────────────────┴─────────╯

Showing task run history

sctool task history <type/task-id> [--limit limit]  [flags]

  • limit - Limit the number of returned results (default: no limit)

Flags: -h, --help - Help for task command

Global Flags: see above

Using sctool task history <type/task-id> a chronological list of past task runs can be displayed. Use the --limit flag to limit the number of returned results. A missing stop time field for a running task is indicated with a minus (“-”) as it has not been set yet. Example:

sctool task history repair/b509ac5d-c10a-48e7-a9f1-a938cb2d1db7
╭──────────────────────────────────────┬──────────────────────────┬──────────────────────────┬─────────╮
│ id                                   │ start time               │ stop time                │ status  │
├──────────────────────────────────────┼──────────────────────────┼──────────────────────────┼─────────┤
│ db0af9a1-e59f-11e7-942a-a2e2a20d7a18 │ 2017-12-20T16:07:19.557Z │ -                        │ running │
│ 567d9fcd-e59f-11e7-9429-a2e2a20d7a18 │ 2017-12-20T16:03:37.172Z │ 2017-12-20T16:05:09.266Z │ stopped │
│ c1aa4202-e599-11e7-9428-a2e2a20d7a18 │ 2017-12-20T15:23:40Z     │ 2017-12-20T15:25:12.08Z  │ stopped │
╰──────────────────────────────────────┴──────────────────────────┴──────────────────────────┴─────────╯

Manually starting/stopping scheduled tasks

sctool task start <type/task-id> [flags]

sctool task stop <type/task-id> [flags]

  • task-id - Task identifier, avilable from sctool task list

Flags: -h, --help - Help for task command

Global Flags: see above

A task can be manually started with the sctool task start type/task-id command. Starting an already running task will first stop the previous run and start a new one. A task can be manually stopped with the sctool task stop type/task-id command. Stopping an already stopped task has no effect.

Automatic repair scheduling

Whenever a Scylla cluster is added to the list of managed cluster, a special task of type repair_auto_schedule is created. It is set to run every 7 days at midnight UTC. When the repair_auto_schedule task runs it ensures all available keyspaces in the managed Scylla cluster have a corresponding Repair Unit defined. It then proceeds to schedule a repair task for each Repair Unit. The scheduled repair tasks are set to run starting at two hours past midnight UTC, with enough retries to last for 6 days.

Environment variables

The SCYLLA_MANAGER_CLUSTER environment variable can be used instead of specifying the cluster name/ID with the --cluster flag on command invocations. The SCYLLA_MANAGER_API_URL environment variable can be used instead of specifying the --api-url flag to indicate the Scylla Manager server listening endpoint (for example if the default was changed from http://localhost:8889/api/v1).