How to Configure Client to Server Encryption (SSL) on a Live Scylla Cluster

This document describes the process on how to switch to SSL encryption mode on an already working Scylla cluster with clients connected to it, without causing any downtime. The second section of this doc describe how to enable matching encryption protocol from the client side.

Steps 1-2 must be executed on every Scylla node you wish to enable SSL encryption mode. These steps can be done on all nodes together or one-by-one. Steps 3-6 should be performed one node at a time (one-by-one), so to allow the cluster to continue serving IO. Steps 7-9 are client side validation steps to demonstrate that clients can now connect in SSL encryption mode.

Procedure

1. Follow the procedure for client-server encryption as documented here. This needs to be done on every Scylla node you wish to enable the client-server SSL encryption mode.

Note: all files you generate must be accessible for Scylla user (e.g. I placed them under /etc/scylla)

  1. Run nodetool drain on each node prior to moving into client-server SSL encrypted mode
  2. Stop Scylla on the node/s that are changed to encrypted mode: sudo systemctl stop scylla
  3. After editing /etc/scylla/scylla.yaml to modify the client encryption options it should look like this:
client_encryption_options:
    enabled: true
    certificate: /etc/scylla/db.crt
    keyfile: /etc/scylla//db.key

Note: params (enabled, certificate, keyfile) must be 4 spaces indent to the header (client_encryption_options:)

  1. Start Scylla: sudo systemctl start scylla
  2. Check Scylla logs journalctl _COMM=scylla for the following message: storage_service - Enabling encrypted CQL connections between client and server

Client side validation steps

7. In order for cqlsh to work in client-server encryption SSL mode, you need to generate cqlshrc file: vi ~/.cassandra/cqlshrc

Example:
[authentication]
username = myusername
password = mypassword
[cql]
; Substitute for the version of Cassandra you are connecting to.
version = 3.3.1
[connection]
hostname = 127.0.0.1
port = 9042
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /etc/scylla/db.crt
; Note: If validate = true then the certificate name must match the machine's hostname
validate = true
; If using client authentication (require_client_auth = true in cassandra.yaml) you'll also need to point to your userkey and usercert.
; SSL client authentication is only supported via cqlsh on C* 2.1 and greater.
; This is disabled by default on all Instaclustr-managed clusters.
userkey = /etc/scylla/db.key
usercert = /etc/scylla/db.crt
  1. Copy the following created files (db.key, db.crt, cadb.key, cadb.pem) to your client/s, from which you run cassandra stress
  2. In order to run cassandra stress using SSL you need to create java key store (jks) from the .pem file on the every client that runs cassandra stress
  • Generate the Java keystore for the server certs
openssl pkcs12 -export -out keystore.p12 -inkey /home/tomer/server_files/db.key -in /home/tomer/server_files/db.crt keytool -importkeystore -  destkeystore keystore.jks -srcstoretype PKCS12 -srckeystore keystore.p12
  • Generate the Java truststore for the trust provider
openssl pkcs12 -export -out truststore.p12 -inkey /home/tomer/server_files/cadb.key -in /home/tomer/server_files/cadb.pem keytool -importkeystore -destkeystore truststore.jks -srcstoretype PKCS12 -srckeystore truststore.p12
  • Download and install the Java security providers:

Install to <jre>/lib/security

Note: make sure you have the latest version from this location.

  • Run Cassandra stress with the parameters below:
cassandra-stress write n=1000000 cl=ONE -node 10.240.0.48 -transport keystore=keystore.jks keystore-password=[password] truststore=truststore.jks truststore-password=[password] -mode native cql3 -pop -rate threads=50

Note: when running cassandra stress you may encounter an exception, if some nodes are still not in client-server SSL encrypted mode, yet the cassandra stress will continue to run and connect only to the servers it can.

Note2: This procedure works as-is for v1.7 or higher. When using Scylla v1.6.x you will need a dummy keystore in the default (conf/.keystore) location with password “cassandra” to run. The contents is irrelevant. Also, it only pertains to cassandra-stress. It has no impact/relation to using the normal java driver connection or cqlsh.

An example for Cassandra 2.x/scylla 1.6 by generating a dummy keystore (with a dummy cert - not used at all): keytool -genkey -keyalg RSA -alias selfsigned -keystore conf./.keystore -storepass cassandra -validity 360 -keysize 2048 in my scylla-tools-java

Security