Add a cluster or a node to Scylla Manager¶
Scylla Manager manages clusters. A cluster contains one or more nodes/datacenters. When you add a cluster to Scylla Manager, it adds all of the nodes which are associated with it.
For all Scylla Nodes, confirm that port :9042 and port :22 are open. Port 9042 is vital for Healthcheck, and port 22 is necessary for Repair.
Scylla Manager communicates with a Scylla cluster using REST API. It’s required that the REST API is accessible by Scylla Manager on all of the cluster nodes. By default, Scylla binds REST API to the localhost, preventing other hosts from having access to it. Scylla Manager communicates with Scylla nodes over SSH, ensuring a secure connection for Scylla Management.
Verify that you have SSH access to all nodes in the cluster as well as Scylla Manager Server.
Verify that the user attached to the SSH key is a
sudo enabled user.
Make sure that you know the path to the private key (which is saved on the Scylla Manager Server).
If you cannot establish an SSH connection, you will need to make sure that:
You have created an SSH key pair
The Private Key is on Scylla Manager
The Public Key is on each Scylla Node
Scylla Manager comes with
scyllamgr_ssh_setup script that helps you with user creation, key generation, and distribution.
There are three stages to the script:
Stage 1 - The SSH connection to the nodes from Scylla Manager is tested using the key, which you used to test connectivity (Prerequisites).
Stage 2 - A new SSH key is made, the private key is stored on Scylla Manager Server, and the Public Key is distributed to each Scylla node in the cluster. A new user (
scylla-managerby default), is created and listed in the SSH key.
Stage 3 - Using the new SSH key, connectivity is tested again, and connectivity to the Scylla REST API is tested as well. Upon completion, a confirmation message is displayed.
scyllamgr_ssh_setup may be invoked many times without affecting Scylla Manager or the nodes. If you generate a new key the old key is overwritten.
From the Scylla Manager server, run
scyllamgr_ssh_setupto configure SSH access to the cluster.
scyllamgr_ssh_setup [--ssh-user <username>] [--ssh-identity-file <path to private key/filename>] [--ssh-config-file <path to SSH config file>] [--create-user <username>] [--single-node] [--debug] SCYLLA_NODE_IP
--ssh-user <username>is the SSH username listed in the public key located on the Scylla nodes. This user must be a sudo enabled user.
--ssh-identity-file <file>is the path to the Private key located on Scylla Manager server.
--create-user <username>is the new username which will be listed in the new public key when it is placed on the Scylla nodes, default user is
--ssh-config-file <file>is the path to an alternate configuration file. See Configure connectivity with a configuration file for an example.
--single-nodeis used to set up only the given node, skipping discovery of any other cluster nodes
--debugis used to display debug information
Done!message displays. Make sure that the new keys which scyllamgr_ssh_setup creates are safely stored.
Proceed to Add a cluster.
In this example, the SSH Private Key is located on Scylla Manager Server in /home/centos/.ssh directory and is named key.pem. The username in the SSH Private Key is centos. When the script runs to connect to Scylla Node 192.0.2.14 (a seed node, but not required), a new SSH key is created with the default user (as –create-user is not used) attached to the key.
scyllamgr_ssh_setup --ssh-user centos --ssh-identity-file /home/centos/.ssh/key.pem 192.0.2.14 > Discovering all the cluster nodes > Creating user 'scylla-manager' 192.0.2.14 OK 192.0.2.15 OK 192.0.2.16 OK > Testing Scylla API connectivity 192.0.2.14 OK 192.0.2.15 OK 192.0.2.16 OK > Done! > To add the cluster to Scylla Manager run: sctool cluster add --host 192.0.2.14 --ssh-user 'scylla-manager' --ssh-identity-file '/home/centos/.ssh/scylla-manager.pem'
In cases where different clusters require different credentials use the
--config-file option to specify host to identity file mapping using SSH configuration file.
For example, there are two datacenters. Each datacenter is running with its own identity configuration (different SSH keys and users). In order for the scyllamgr_ssh_setup script to connect to each datacenter, you can use a configuration file that will map the credentials to the correct machines. This configuration file can be placed in any location, but by default, it is read from ~/.ssh/config.
If the configuration file is saved in ~/.ssh/config it will be used by every SSH command. If you want to use this configuration only for Scylla, consider saving the file in another location.
Given an SSH configuration file named config, saved in ~/.ssh/scylla_config:
Host 198.100.51.* User ec2-user IdentityFile /path/to/ec2-user.pem Host 10.2.* User centos IdentityFile /path/to/centos.pem
scyllamgr_ssh_setup --ssh-config-file ~/.ssh/scylla_config config 192.0.2.14 > Discovering all the cluster nodes > Creating user 'scylla-manager' 192.0.2.14 OK 192.0.2.15 OK 192.0.2.16 OK > Testing Scylla API connectivity 192.0.2.14 OK 192.0.2.15 OK 192.0.2.16 OK > Done! > To add the cluster to Scylla Manager run: sctool cluster add --host 192.0.2.14 --ssh-user 'scylla-manager' --ssh-identity-file '/home/centos/.ssh/scylla-manager.pem'
The SSH configuration file parameter may always be used to specify additional SSH options.
man ssh_config for information on file format and available options.
Continue to Add a cluster.
If you encounter trouble setting up SSH connectivity to a node, use the
scyllamgr_ssh_setup command with
--debug flag to show debug information.
If you have setup SSH communication with the cluster as described in the previous section, please continue to Add a cluster. If you have not used the procedure in Enable SSH Communication to establish SSH, you will need to create a connection manually.
exposing Scylla REST API may have security consequences. This procedure is not recommended.
Scylla Manager can interact with Scylla REST API on cluster nodes directly. For that configuration of all the cluster nodes must be adjusted, and the nodes must be restarted.
For every node in the cluster, repeat the procedure below.
SSH to a node.
Edit Scylla config file
api_addressto the broadcast address.
Drain the node (stop accepting writes and flush all tables).
sudo systemctl restart scylla-server.service
Verify the Scylla server is running.
Before you begin
From the Scylla Manager Server, you can copy and paste the confirmation message you received when you connected your cluster with scyllamgr_ssh_setup , which instructs you to run:
sctool cluster add.
sctool cluster add --host 192.0.2.14 --ssh-user 'scylla-manager' --ssh-identity-file '/home/centos/.ssh/scylla-manager.pem' --name manager-testcluster 085bdfe2-91c7-4f6c-aeb0-cfea107ff9f0 __ / \ Cluster added, to set it as a default run: @ @ export SCYLLA_MANAGER_CLUSTER=085bdfe2-91c7-4f6c-aeb0-cfea107ff9f0 | | || |/ Repair will run on 03 Apr 19 00:00:00 UTC and will be repeated every 7d. || || To see the currently scheduled tasks: sctool task list -c 085bdfe2-91c7-4f6c-aeb0-cfea107ff9f0 |\_/| \___/
--hostis hostname or IP of one of the cluster nodes
--nameis an alias you can give to your cluster. Using an alias means you do not need to use the ID of the cluster in all other operations.
--ssh-useris SSH user name used in the
--create-user <username>field when you ran the scyllamgr_ssh_setup
--ssh-identity-filepath is a path to identity file containing SSH private key
If you exposed the REST API directly to establish SSH connectivity (Direct Communication), the
--ssh-identity-file parameters should be omitted.
Each cluster has a unique ID. You will use this ID in all commands where the cluster ID is required. Each cluster is automatically registered with a repair task that runs once a week.
Verify that the cluster you added has a registered repair task by running the
sctool task listcommand, adding the name of the cluster (using the parameter
-c manager-testcluster) you created in step 1.
sctool task list -c manager-testcluster Cluster: manager-testcluster ╭──────────────────────────────────────────────────────┬───────────────────────────────┬──────┬───────────┬────────╮ │ task │ next run │ ret. │ arguments │ status │ ├──────────────────────────────────────────────────────┼───────────────────────────────┼──────┼───────────┼────────┤ │ healthcheck/018da854-b9ff-4e0a-bae7-ca65c677c559 │ 02 Apr 19 18:06:31 UTC (+15s) │ 0 │ │ NEW │ │ healthcheck_api/597f237f-103d-4994-8167-3ff591150b7e │ 02 Apr 19 18:07:01 UTC (+1h) │ 0 │ │ NEW │ │ repair/21006f88-0c8c-4e11-9e84-83c319f80d0c │ 03 Apr 19 00:00:00 UTC (+7d) │ 3/3 │ │ NEW │ ╰──────────────────────────────────────────────────────┴───────────────────────────────┴──────┴───────────┴────────╯
You will see 3 tasks which are created by adding the cluster:
Healthcheck - which checks the Scylla CQL, starting immediately, repeating every 15 seconds. See Scylla Health Check
Healthcheck API - which checks the Scylla REST API, starting immediately, repeating every hour. See Scylla Health Check
Repair - an automated repair task, starting at midnight tonight, repeating every seven days at midnight. See Run a Repair
If you want to change the schedule for the repair, see Reschedule a repair.
Connecting your cluster to Scylla Monitoring allows you to see metrics about your cluster and Scylla Manager all within Scylla Monitoring.
If you have any issues connecting to Scylla Monitoring Stack, consult the Troubleshooting Guide.
Although Scylla Manager is aware of all topology changes made within every cluster it manages, it cannot properly manage nodes/datacenters without establishing SSH connections with every node/datacenter in the cluster. Keys cannot be automatically distributed, so to make sure this connection is established, you need to run the scyllamgr_ssh_setup.
Before You Begin
Confirm you have a managed cluster running under Scylla Manager. If you do not have a managed cluster, see Add a cluster.
Run the procedure as described in Configure Scylla Manager connectivity using scyllamgr_ssh_setup above. If you are adding a single node, you can leave out the
--discoverflag. If you are adding multiple nodes (or a Datacenter), you may either specify them all individually or use the
--discoverflag. Make sure you receive a confirmation with the IP address of the new node. For example:
Confirm the node/datacenter was added by checking its status from the node running Scylla Manager server run the
sctool status --cluster <cluster-name>command, using the name of the managed cluster.
sctool status -c manager-testcluster Datacenter: us-east ╭──────────┬──────────┬─────┬──────────────╮ │ CQL │ API │ SSL │ Host │ ├──────────┼──────────┼─────┼──────────────┤ │ UP (0ms) │ UP (0ms) │ OFF │ 18.104.22.168 │ │ UP (0ms) │ UP (0ms) │ OFF │ 22.214.171.124 │ │ UP (0ms) │ UP (0ms) │ OFF │ 126.96.36.199 │ │ UP (0ms) │ UP (0ms) │ OFF │ 188.8.131.52 │ ╰──────────┴──────────┴─────┴──────────────╯
If you are using the Scylla Monitoring Stack, continue to Connect Managed Cluster to Scylla Monitoring for more information.
There is no need to perform any action in Scylla Manager after removing a node or datacenter from a Scylla cluster.
If you are removing the cluster from Scylla Manager and you are using Scylla Monitoring, refer to Prometheus Target List for more information.
If you deleted the SSH key files, they could be restored.
From an existing node search the
/var/lib/scylla-manager/.ssh/authorized_keys directory and look for lines under the comment
# Added by Scylla Manager. There you will find a copy of the public key. You can copy this key to connect to the private key held by Scylla Manager.